Oct 30 19:13:43.571799 ip-10-0-138-233 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:13:43.571812 ip-10-0-138-233 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:13:43.571819 ip-10-0-138-233 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:13:43.572090 ip-10-0-138-233 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:13:54.796542 ip-10-0-138-233 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:13:54.796558 ip-10-0-138-233 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4b71749228084cc9b74086026f26beef -- Oct 30 19:15:58.227352 ip-10-0-138-233 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:15:58.620211 ip-10-0-138-233 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:58.620211 ip-10-0-138-233 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:15:58.620211 ip-10-0-138-233 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:58.620709 ip-10-0-138-233 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:58.625195 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625014 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.625195 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625194 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.625195 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625202 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625205 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625208 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625211 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625214 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625217 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625222 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625225 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625229 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625231 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625234 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625237 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625240 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625243 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625245 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625248 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625250 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625253 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.625347 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625261 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625265 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625268 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625278 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625281 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625284 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625287 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625289 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625292 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625312 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625315 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625317 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625320 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625323 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625327 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625333 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625337 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625339 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.625794 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625342 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625345 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625348 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625351 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625353 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625356 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625359 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625361 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625364 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625366 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625369 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625372 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625375 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625380 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625384 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625387 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625390 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625393 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625397 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625399 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.626258 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625402 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625404 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625407 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625410 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625412 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625415 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625418 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625421 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625424 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625427 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625429 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625432 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625435 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625437 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625440 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625444 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625448 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625451 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625453 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625456 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.626728 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625458 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625461 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625465 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625468 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625471 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625473 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625476 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625478 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625481 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625483 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625486 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625489 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625491 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625494 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625496 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625499 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625502 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625505 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625509 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.627237 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625511 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625517 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625520 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625523 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625526 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.625529 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627062 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627069 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627072 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627090 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627094 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627097 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627100 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627102 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627105 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627108 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627111 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.627667 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627116 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627118 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627121 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627123 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627126 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627128 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627131 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627133 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627136 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627138 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627141 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627143 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627145 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627149 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627152 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627154 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627157 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627160 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627162 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.628097 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627164 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627167 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627170 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627172 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627175 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627177 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627180 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627182 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627185 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627187 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627189 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627192 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627195 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627197 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627200 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627202 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627205 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627207 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627210 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627213 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.628553 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627215 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627218 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627220 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627223 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627225 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627227 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627233 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627236 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627239 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627242 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627245 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627248 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627250 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627253 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627255 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627258 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627260 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627262 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627265 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.629002 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627268 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627270 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627272 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627275 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627277 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627280 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627283 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627287 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627290 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627293 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627296 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627298 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627301 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627303 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627306 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627308 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627310 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627313 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627315 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.629507 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627318 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627321 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627323 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627326 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627329 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627331 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627334 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627336 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627339 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627341 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627344 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627347 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627349 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627352 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627354 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627356 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627833 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627837 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627841 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627844 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.629972 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627846 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627849 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627851 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627854 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627857 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627859 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627862 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627867 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627870 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627872 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627875 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627878 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627880 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627884 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627888 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627890 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627893 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627896 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.630456 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627898 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627900 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627903 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627905 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627908 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627910 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627914 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627917 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627920 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627922 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627925 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627928 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627931 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627933 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627936 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627938 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627941 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627944 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627946 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.630903 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627954 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627957 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627960 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627962 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627964 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627967 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627970 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627972 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627975 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627977 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627980 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627982 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627985 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627987 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627990 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627992 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627994 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.627999 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.631364 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628002 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628004 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628007 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628009 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628011 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628014 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628017 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628019 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628022 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628024 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628027 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628029 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628032 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628036 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628038 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628041 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628043 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628046 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628050 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628053 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.631838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628055 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628057 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628060 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628063 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628065 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628068 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628070 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628073 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628088 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628090 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628093 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628095 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628098 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628101 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628103 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628106 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628108 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628111 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628114 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628119 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.632308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628121 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628124 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628126 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628128 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628211 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628219 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628225 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628229 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628235 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628238 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628257 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628261 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628264 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628267 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628271 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628274 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628277 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628279 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628282 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628285 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628288 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628291 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628294 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:15:58.632838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628297 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628300 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628303 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628306 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628310 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628313 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628315 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628319 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628321 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628327 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628330 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628334 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628337 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628341 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628344 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628346 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628349 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628352 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628355 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628360 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628363 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628366 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628369 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628372 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:15:58.633393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628376 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628379 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628382 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628384 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628388 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628391 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628394 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628397 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628399 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628402 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628405 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628409 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628412 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628415 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628418 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628421 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628424 2579 flags.go:64] FLAG: --help="false" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628427 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628431 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628434 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628437 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628440 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628443 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:15:58.634001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628446 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628449 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628451 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628454 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628459 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628462 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628465 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628468 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628470 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628473 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628476 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628479 2579 flags.go:64] FLAG: --lock-file="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628482 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628484 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628487 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628492 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628495 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628498 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628501 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628503 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628506 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628509 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628512 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628516 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628519 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:15:58.634573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628522 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628525 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628528 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628531 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628534 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628537 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628540 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628543 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628549 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628552 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628555 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628559 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628562 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628565 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628568 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628572 2579 flags.go:64] FLAG: --port="10250" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628575 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628577 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0715b9e8fe625cdcd" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628580 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628583 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628586 2579 flags.go:64] FLAG: --register-node="true" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628589 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628593 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628595 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628598 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628601 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:15:58.635175 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628604 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628607 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628610 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628612 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628615 2579 flags.go:64] FLAG: --runonce="false" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628618 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628621 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628624 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628627 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628629 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628633 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628638 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628641 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628643 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628646 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628649 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628652 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628656 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628659 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628662 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628667 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628669 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628672 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628676 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628678 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:15:58.635762 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628681 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628684 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628687 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628689 2579 flags.go:64] FLAG: --v="2" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628693 2579 flags.go:64] FLAG: --version="false" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628697 2579 flags.go:64] FLAG: --vmodule="" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628701 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.628704 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628826 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628830 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628834 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628837 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628840 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628843 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628846 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628848 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628851 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628853 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628858 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628860 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628863 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.636341 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628866 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628868 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628870 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628874 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628877 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628879 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628882 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628884 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628887 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628890 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628892 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628895 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628897 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628899 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628902 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628905 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628907 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628910 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628912 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628914 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.636822 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628917 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628919 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628922 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628924 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628927 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628929 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628931 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628934 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628936 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628942 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628946 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628949 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628952 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628955 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628958 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628961 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628964 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628967 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628969 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.637322 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628972 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628977 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628979 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628982 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628984 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628987 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628989 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628992 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628994 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628996 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.628999 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629002 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629004 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629007 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629010 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629012 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629015 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629017 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629020 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.637773 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629023 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629025 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629029 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629032 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629035 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629037 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629040 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629042 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629046 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629048 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629051 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629053 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629055 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629058 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629060 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629063 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629065 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629068 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629070 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629073 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.638253 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629087 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629089 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629092 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629095 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629097 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629100 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629102 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629105 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629108 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629110 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629115 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629118 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629120 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.629148 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629184 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.638724 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629193 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629196 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629199 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629203 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629206 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629211 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629213 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629216 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629218 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629221 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629223 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629226 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629229 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629231 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629234 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629236 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629239 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.639136 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629242 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629244 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629247 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629249 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629252 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629255 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629265 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629268 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629271 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629273 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629276 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629278 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629282 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629285 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629287 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629308 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629312 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629315 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629318 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629321 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.639574 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629323 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629326 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629328 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629331 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629334 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629336 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629339 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629341 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629344 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629346 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629349 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629351 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629354 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629356 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629359 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629361 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629364 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629367 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629369 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.640016 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629372 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629374 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629378 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629381 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629384 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629389 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629393 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629396 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629399 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629402 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629410 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629412 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629416 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629418 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629421 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629424 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629426 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629429 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629431 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.640490 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629434 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629436 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629440 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629442 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629445 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629448 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629450 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629453 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629455 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629458 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629460 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629463 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629465 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629468 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629471 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629473 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629476 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629478 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629482 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629485 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.640912 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629487 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629490 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629492 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629496 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629500 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629504 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629508 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.629512 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.635424 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.635437 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.635453 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.635458 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635512 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635517 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635520 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635523 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635526 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635528 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635531 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.641409 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635534 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635537 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635539 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635542 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635544 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635547 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635549 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635552 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635554 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635557 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635559 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635562 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635565 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635567 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635570 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635572 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635575 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635580 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635583 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.641865 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635586 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635589 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635592 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635594 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635597 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635600 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635603 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635605 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635608 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635611 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635613 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635615 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635618 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635620 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635623 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635626 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635628 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635632 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635635 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.642326 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635638 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635641 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635644 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635646 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635649 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635651 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635654 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635656 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635658 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635661 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635664 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635667 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635670 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635673 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635675 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635678 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635681 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635684 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635686 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635689 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.643050 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635691 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635694 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635696 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635699 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635701 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635704 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635707 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635709 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635712 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635714 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635717 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635719 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635722 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635724 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635727 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635729 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635731 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635734 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635737 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635739 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.643762 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635742 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635744 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635747 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635750 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635754 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635756 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635758 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635761 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635764 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635766 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635769 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635772 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635774 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635777 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635780 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635782 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635785 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635787 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635790 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635792 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.644328 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635794 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.635819 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635949 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635954 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635957 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635959 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635962 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635965 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635968 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635971 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635975 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635977 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635980 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635983 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635985 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635988 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.644784 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635991 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635994 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635996 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.635999 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636002 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636004 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636007 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636009 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636012 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636014 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636017 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636019 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636021 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636024 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636026 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636028 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636031 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636033 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636036 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636038 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.645177 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636040 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636043 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636045 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636048 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636050 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636052 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636055 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636057 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636060 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636062 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636065 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636067 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636070 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636090 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636093 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636096 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636098 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636100 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.645675 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636104 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636106 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636109 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636111 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636116 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636118 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636121 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636123 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636126 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636128 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636130 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636133 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636135 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636138 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636140 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636142 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636145 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636147 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.646401 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636150 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636153 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636156 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636159 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636161 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636164 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636166 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636169 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636171 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636174 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636177 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636180 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636183 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636186 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636188 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636190 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636193 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636195 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636197 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.646880 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636200 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636202 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636204 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636207 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636209 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636212 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636214 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636216 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636221 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636224 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636226 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636228 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636231 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636233 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.647435 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.636256 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636287 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636292 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636295 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636298 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636301 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636304 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636307 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636310 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636313 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636316 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636318 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636321 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636323 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636326 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636328 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636331 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636333 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636336 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:58.647847 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636338 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636341 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636344 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636346 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636349 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636351 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636354 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636357 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636359 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636362 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636364 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636367 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636369 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636374 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636376 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636379 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636381 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636385 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636387 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636390 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:58.648421 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636392 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636395 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636398 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636401 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636405 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636408 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636410 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636413 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636415 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636418 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636420 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636423 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636425 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636428 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636430 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636432 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636435 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636437 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:58.648866 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636440 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636442 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636445 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636447 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636449 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636452 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636456 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636458 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636461 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636465 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636468 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636471 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636474 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636476 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636480 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636483 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636486 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636488 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636491 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:58.649370 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636493 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636496 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636498 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636501 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636503 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636506 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636508 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636511 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636513 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636515 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636518 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636520 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636523 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636525 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636528 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636530 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636533 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636537 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636539 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:58.649809 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636543 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636546 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636548 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636551 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636553 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636556 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636558 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636561 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636564 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636567 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:15:58.636569 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.636615 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.638448 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.639458 2579 server.go:1014] "Starting client certificate rotation" Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.639550 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.640236 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.641491 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:58.650311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.643985 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:58.652794 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.652778 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:15:58.655925 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.655909 2579 log.go:25] "Validated CRI v1 image API" Oct 30 19:15:58.656726 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.656706 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:15:58.660657 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.660640 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b04e1b63-d6a8-4d39-a43f-bd79372f3797:/dev/nvme0n1p3 d604349d-da9e-42eb-bab5-7290335ac519:/dev/nvme0n1p4] Oct 30 19:15:58.660699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.660657 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:15:58.665493 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.665386 2579 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:58.664149455 +0000 UTC m=+0.320840164 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3175369 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c5517c0a9a2d3769de8f313f2a7f SystemUUID:ec23c551-7c0a-9a2d-3769-de8f313f2a7f BootID:4b717492-2808-4cc9-b740-86026f26beef Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:94:42:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:94:42:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:0d:7c:05:56:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:15:58.665493 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.665491 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:15:58.665601 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.665585 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:15:58.669315 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.669269 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:15:58.669466 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.669317 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-233.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:15:58.669977 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.669964 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:15:58.670035 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.669981 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:15:58.670035 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.670023 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:58.670127 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.670041 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:58.670939 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.670927 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:15:58.673904 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.673884 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:15:58.674724 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674712 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:15:58.674859 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674848 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:15:58.674920 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674886 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:15:58.674920 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674906 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:15:58.674920 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674919 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:15:58.675028 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674931 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:15:58.675028 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.674941 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:15:58.675253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.675240 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:58.678743 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.678719 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:15:58.683860 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.683840 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:58.684646 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684630 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:15:58.684699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684676 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:15:58.684864 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684852 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:15:58.684894 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684870 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:15:58.684894 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684876 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:15:58.684894 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684882 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:15:58.684894 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684887 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:15:58.684894 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.684892 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:15:58.685405 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.685395 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:15:58.685405 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.685404 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:15:58.685458 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.685412 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:15:58.685458 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.685418 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:15:58.685458 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.685427 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:15:58.685458 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.685437 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:15:58.686107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.686097 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:15:58.686156 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.686108 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:15:58.690117 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.690103 2579 server.go:1263] "Started kubelet" Oct 30 19:15:58.690234 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.690195 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:15:58.690283 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.690249 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:15:58.690432 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.690417 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-233.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:15:58.690485 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.690458 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:15:58.690748 ip-10-0-138-233 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:15:58.691273 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.691254 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:15:58.691979 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.691965 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:15:58.693315 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.693204 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7kq7g" Oct 30 19:15:58.695366 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.695344 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:15:58.696492 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.696469 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:15:58.696566 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.696469 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-233.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:15:58.696901 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.696887 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:15:58.697839 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.697823 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:15:58.697959 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.697941 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:58.698674 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.698368 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:15:58.698674 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.698382 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:15:58.698674 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.698448 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:15:58.698674 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.698531 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:15:58.698674 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.698538 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:15:58.699376 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.699359 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:15:58.699376 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.699375 2579 factory.go:55] Registering systemd factory Oct 30 19:15:58.699376 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.699383 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:15:58.700764 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.700748 2579 factory.go:153] Registering CRI-O factory Oct 30 19:15:58.700764 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.700765 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 19:15:58.700905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.700780 2579 factory.go:103] Registering Raw factory Oct 30 19:15:58.700905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.700794 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 19:15:58.701575 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.701526 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-233.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:15:58.701661 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.700845 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-233.us-east-2.compute.internal.18e364aa2d414c6f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-233.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-233.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:58.690061423 +0000 UTC m=+0.346752132,LastTimestamp:2026-10-30 19:15:58.690061423 +0000 UTC m=+0.346752132,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-233.us-east-2.compute.internal,}" Oct 30 19:15:58.701729 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.701663 2579 manager.go:319] Starting recovery of all containers Oct 30 19:15:58.702945 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.702921 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7kq7g" Oct 30 19:15:58.704690 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.704668 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:15:58.714312 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.714296 2579 manager.go:324] Recovery completed Oct 30 19:15:58.719509 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.719495 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.721269 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721248 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.721370 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721280 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.721370 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721291 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.721888 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721874 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:15:58.721888 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721888 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:15:58.721972 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721875 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.721972 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.721905 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:15:58.722604 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.722589 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.722676 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.722613 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.722676 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.722622 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.723913 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.723902 2579 policy_none.go:50] "Start" Oct 30 19:15:58.723958 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.723920 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:15:58.723958 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.723933 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:15:58.725935 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.725923 2579 policy_none.go:44] "Start" Oct 30 19:15:58.751710 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.751689 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:15:58.754747 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.754733 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 19:15:58.755259 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.755245 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:15:58.755259 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.755260 2579 server.go:87] "Starting device plugin registration server" Oct 30 19:15:58.755440 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.755430 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:15:58.755481 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.755441 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:15:58.755584 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.755505 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:15:58.755651 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.755590 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:15:58.755651 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.755598 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:15:58.756145 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.756126 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:15:58.756208 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.756170 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:58.841288 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.841267 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:15:58.843228 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.842353 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:15:58.843228 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.842375 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:15:58.843228 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.842393 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:15:58.843228 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.842429 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:15:58.846791 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.846767 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:58.855877 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.855863 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.856573 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.856554 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.856648 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.856590 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.856648 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.856601 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.856648 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.856623 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.865467 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.865453 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.865520 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.865473 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-233.us-east-2.compute.internal\": node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:58.884353 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.884293 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:58.943379 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.943358 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"] Oct 30 19:15:58.943448 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.943403 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.944161 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.944147 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.944212 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.944177 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.944212 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.944191 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.946332 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.946320 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.946481 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.946468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.946516 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.946501 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.947003 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.946986 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.947103 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.947013 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.947103 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.947028 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.947103 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.947040 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.947103 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.947058 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.947103 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.947067 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.949239 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.949221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.949359 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.949245 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:58.949968 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.949952 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:58.950050 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.949979 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:58.950050 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:58.949995 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:58.976441 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.976425 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.978444 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.978429 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:58.984727 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:58.984712 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:59.000386 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.000360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.000466 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.000392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.000466 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.000418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.085576 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:59.085555 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:59.100880 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.100861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.100934 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.100883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.100934 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.100903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.101002 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.100933 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.101002 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.100947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.101002 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.100959 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.186310 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:59.186260 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:59.280742 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.280722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.282867 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:15:59.282855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:15:59.286700 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:59.286679 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:59.387786 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:59.387768 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:15:59.488255 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:15:59.488214 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:16:00.585037 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.585021 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:16:00.619125 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.619106 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:16:00.644703 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.644687 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:16:00.668758 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.668741 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:16:00.669075 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.668822 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:16:00.669075 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.668825 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:16:00.669075 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.668882 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:16:00.705075 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.705055 2579 apiserver.go:52] "Watching apiserver" Oct 30 19:16:00.720743 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.720725 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:16:00.721001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.720982 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-p74tq","openshift-image-registry/node-ca-mkrdd","openshift-multus/multus-additional-cni-plugins-xbzbq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv","openshift-cluster-node-tuning-operator/tuned-ddx75","openshift-network-diagnostics/network-check-target-tfj5g","openshift-multus/multus-w2lmc","openshift-multus/network-metrics-daemon-lc88s","openshift-network-operator/iptables-alerter-mwhv4","openshift-ovn-kubernetes/ovnkube-node-p682z","kube-system/global-pull-secret-syncer-5sqtd","kube-system/konnectivity-agent-9ssdd"] Oct 30 19:16:00.723588 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.723568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.725221 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.725202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.726777 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.726762 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:16:00.726913 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.726892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.727007 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.726985 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.727420 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.727378 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6xn8\"" Oct 30 19:16:00.728207 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.727852 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.728207 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728156 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzjv5\"" Oct 30 19:16:00.728356 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728217 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.728356 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728269 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:16:00.728475 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728459 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:16:00.728475 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728471 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:16:00.730178 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728486 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:16:00.730178 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.728694 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.730178 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.729623 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrdg\"" Oct 30 19:16:00.730178 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.729648 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.730178 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.729697 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.730418 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.730404 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:16:00.730953 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.730939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733142 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-kubernetes\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733225 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-var-lib-kubelet\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733225 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9t4\" (UniqueName: \"kubernetes.io/projected/3489239b-c17e-41ac-84f3-392b94d019e5-kube-api-access-sc9t4\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733225 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:00.733225 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-system-cni-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.733413 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.733413 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.733273 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:00.733413 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733413 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-host\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733555 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.733555 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-systemd\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733555 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhpt\" (UniqueName: \"kubernetes.io/projected/9ef62b57-79b3-4002-ba56-d41d3b74ded9-kube-api-access-2nhpt\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.733555 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-device-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-run\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-tmp\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cnibin\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.733818 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.733818 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-sys-fs\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733818 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jgh\" (UniqueName: \"kubernetes.io/projected/53f48b9b-ec52-44f6-a537-e6e585762c7e-kube-api-access-46jgh\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733818 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-conf\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733992 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-socket-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733992 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-registration-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.733992 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-sys\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.733992 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-hosts-file\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.733992 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qf2c\" (UniqueName: \"kubernetes.io/projected/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-kube-api-access-6qf2c\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.733992 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysconfig\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.734189 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.733998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.734189 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.734057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-etc-tuned\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.734189 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.734077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-tmp-dir\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.734189 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.734124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-modprobe-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.734189 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.734150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-lib-modules\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.734189 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.734186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.735137 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.735117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.737538 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.737521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.737765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.737713 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:59 +0000 UTC" deadline="2028-05-01 22:27:21.985454612 +0000 UTC" Oct 30 19:16:00.737832 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.737765 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13179h11m21.247692807s" Oct 30 19:16:00.738798 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.738784 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:16:00.738889 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.738878 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:16:00.739507 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.739494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.741413 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.741402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.743367 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.743353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.745393 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.745379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:00.745481 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.745446 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:00.745885 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.745872 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:16:00.747327 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.747310 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"] Oct 30 19:16:00.747390 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.747341 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal"] Oct 30 19:16:00.747424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.747400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.747451 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.747442 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:00.756619 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.756602 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:16:00.756697 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.756659 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qv2vg" Oct 30 19:16:00.768370 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.768332 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qv2vg" Oct 30 19:16:00.827495 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827276 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:16:00.827495 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827486 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.827722 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827322 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.827722 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827325 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:16:00.827722 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827305 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.827871 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827761 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:16:00.827923 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.827907 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:16:00.828844 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.828827 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:16:00.828938 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.828911 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.828938 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.828911 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wctlt\"" Oct 30 19:16:00.829036 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.828940 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qmpzj\"" Oct 30 19:16:00.829036 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.828941 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:16:00.829036 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.828992 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.829036 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.829036 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829021 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:16:00.829262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829134 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:16:00.829262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829138 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mglp5\"" Oct 30 19:16:00.829262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829148 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7s42\"" Oct 30 19:16:00.829262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829182 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.829262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829192 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.829262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829186 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xph5z\"" Oct 30 19:16:00.829626 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829575 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:16:00.829967 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829952 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:16:00.830034 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.829994 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9tgzd\"" Oct 30 19:16:00.834514 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.834633 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.834633 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-script-lib\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.834633 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-multus\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.834633 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-host\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.834633 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.834633 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79323c8f-776a-4a3e-942c-a467e8672c53-ovn-node-metrics-cert\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-host\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26bc4ca2-8079-4560-8891-dc52f796c72f-agent-certs\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-systemd\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.834928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpt\" (UniqueName: \"kubernetes.io/projected/9ef62b57-79b3-4002-ba56-d41d3b74ded9-kube-api-access-2nhpt\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-systemd\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-systemd-units\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.834998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-run\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-tmp\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cnibin\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-run\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-sys-fs\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jgh\" (UniqueName: \"kubernetes.io/projected/53f48b9b-ec52-44f6-a537-e6e585762c7e-kube-api-access-46jgh\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-log-socket\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835236 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cnibin\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2pn\" (UniqueName: \"kubernetes.io/projected/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-kube-api-access-tw2pn\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.835253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835252 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-sys-fs\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-conf\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835354 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-conf\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-registration-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835400 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-etc-kubernetes\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cthhb\" (UniqueName: \"kubernetes.io/projected/862b83b6-5666-4f55-a276-4cb674c0b4a8-kube-api-access-cthhb\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835463 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-registration-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgj7\" (UniqueName: \"kubernetes.io/projected/06348567-413f-417b-a78b-a10d7ff83ff4-kube-api-access-9sgj7\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-kubelet-config\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-sys\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-hosts-file\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835580 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835588 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-sys\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qf2c\" (UniqueName: \"kubernetes.io/projected/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-kube-api-access-6qf2c\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dn2\" (UniqueName: \"kubernetes.io/projected/15140033-be3b-4729-9a78-3a38b3d1ef48-kube-api-access-v4dn2\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.835895 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835685 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-hosts-file\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-slash\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-var-lib-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-etc-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-ovn\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.835983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-tmp-dir\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-netd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbpg\" (UniqueName: \"kubernetes.io/projected/79323c8f-776a-4a3e-942c-a467e8672c53-kube-api-access-drbpg\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-host\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-cnibin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-modprobe-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-lib-modules\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-tmp-dir\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-serviceca\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.836465 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-system-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-cni-binary-copy\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-modprobe-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-netns\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-daemon-config\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-lib-modules\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9t4\" (UniqueName: \"kubernetes.io/projected/3489239b-c17e-41ac-84f3-392b94d019e5-kube-api-access-sc9t4\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-system-cni-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-node-log\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836610 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-system-cni-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-socket-dir-parent\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-dbus\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-env-overrides\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-device-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-config\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:00.837107 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15140033-be3b-4729-9a78-3a38b3d1ef48-iptables-alerter-script\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-device-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15140033-be3b-4729-9a78-3a38b3d1ef48-host-slash\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836932 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-systemd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.836963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-k8s-cni-cncf-io\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-socket-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-bin\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-socket-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26bc4ca2-8079-4560-8891-dc52f796c72f-konnectivity-ca\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-bin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-conf-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysconfig\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-etc-tuned\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysconfig\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.837765 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-kubelet\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-kubelet\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-hostroot\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-netns\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-multus-certs\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-kubernetes\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-var-lib-kubelet\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-var-lib-kubelet\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.838391 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.837947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-kubernetes\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.839734 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.839717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-tmp\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.839798 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.839755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-etc-tuned\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.844449 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.844421 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9t4\" (UniqueName: \"kubernetes.io/projected/3489239b-c17e-41ac-84f3-392b94d019e5-kube-api-access-sc9t4\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:00.844563 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.844425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qf2c\" (UniqueName: \"kubernetes.io/projected/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-kube-api-access-6qf2c\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:00.845145 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.845116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jgh\" (UniqueName: \"kubernetes.io/projected/53f48b9b-ec52-44f6-a537-e6e585762c7e-kube-api-access-46jgh\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:00.846270 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.846249 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhpt\" (UniqueName: \"kubernetes.io/projected/9ef62b57-79b3-4002-ba56-d41d3b74ded9-kube-api-access-2nhpt\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:00.916038 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.916018 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:16:00.938475 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-script-lib\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938562 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-multus\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.938562 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79323c8f-776a-4a3e-942c-a467e8672c53-ovn-node-metrics-cert\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938562 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26bc4ca2-8079-4560-8891-dc52f796c72f-agent-certs\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.938562 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-multus\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-systemd-units\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938610 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-systemd-units\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.938694 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-log-socket\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.938760 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.438741246 +0000 UTC m=+2.065646260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:00.938782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-log-socket\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2pn\" (UniqueName: \"kubernetes.io/projected/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-kube-api-access-tw2pn\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-etc-kubernetes\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthhb\" (UniqueName: \"kubernetes.io/projected/862b83b6-5666-4f55-a276-4cb674c0b4a8-kube-api-access-cthhb\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgj7\" (UniqueName: \"kubernetes.io/projected/06348567-413f-417b-a78b-a10d7ff83ff4-kube-api-access-9sgj7\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-kubelet-config\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dn2\" (UniqueName: \"kubernetes.io/projected/15140033-be3b-4729-9a78-3a38b3d1ef48-kube-api-access-v4dn2\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.938980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-slash\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-etc-kubernetes\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-var-lib-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-etc-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-kubelet-config\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-ovn\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-slash\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-ovn\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939324 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939207 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-var-lib-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-netd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbpg\" (UniqueName: \"kubernetes.io/projected/79323c8f-776a-4a3e-942c-a467e8672c53-kube-api-access-drbpg\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-etc-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-host\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939291 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-host\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939364 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-netd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-cnibin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-serviceca\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-system-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939449 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-cnibin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-system-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-cni-binary-copy\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-netns\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-daemon-config\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-node-log\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-socket-dir-parent\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.939883 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939728 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-netns\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-dbus\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939753 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-node-log\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-socket-dir-parent\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-env-overrides\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-dbus\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-config\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15140033-be3b-4729-9a78-3a38b3d1ef48-iptables-alerter-script\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15140033-be3b-4729-9a78-3a38b3d1ef48-host-slash\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-systemd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-k8s-cni-cncf-io\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.939997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-bin\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-systemd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26bc4ca2-8079-4560-8891-dc52f796c72f-konnectivity-ca\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.940686 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-bin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-conf-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-kubelet\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-kubelet\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-hostroot\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-netns\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-multus-certs\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940349 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-script-lib\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-multus-certs\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940372 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-env-overrides\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940422 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-k8s-cni-cncf-io\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940430 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-kubelet\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-hostroot\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940519 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15140033-be3b-4729-9a78-3a38b3d1ef48-host-slash\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-netns\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.940587 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:00.941361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-bin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.940643 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.44062686 +0000 UTC m=+2.067531882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-kubelet\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940638 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-conf-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-bin\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26bc4ca2-8079-4560-8891-dc52f796c72f-konnectivity-ca\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.940943 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15140033-be3b-4729-9a78-3a38b3d1ef48-iptables-alerter-script\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.941075 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-serviceca\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.941117 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26bc4ca2-8079-4560-8891-dc52f796c72f-agent-certs\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.941208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-daemon-config\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.941392 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79323c8f-776a-4a3e-942c-a467e8672c53-ovn-node-metrics-cert\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.941849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.941454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-cni-binary-copy\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.942271 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.942088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-config\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.946310 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.946291 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:00.946401 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.946315 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:00.946401 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.946328 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:00.946401 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:00.946373 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.446357825 +0000 UTC m=+2.073262842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:00.947570 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.947541 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2pn\" (UniqueName: \"kubernetes.io/projected/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-kube-api-access-tw2pn\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:00.948055 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.948032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgj7\" (UniqueName: \"kubernetes.io/projected/06348567-413f-417b-a78b-a10d7ff83ff4-kube-api-access-9sgj7\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:00.948230 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.948214 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dn2\" (UniqueName: \"kubernetes.io/projected/15140033-be3b-4729-9a78-3a38b3d1ef48-kube-api-access-v4dn2\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:00.948349 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.948329 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthhb\" (UniqueName: \"kubernetes.io/projected/862b83b6-5666-4f55-a276-4cb674c0b4a8-kube-api-access-cthhb\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:16:00.948736 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.948717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbpg\" (UniqueName: \"kubernetes.io/projected/79323c8f-776a-4a3e-942c-a467e8672c53-kube-api-access-drbpg\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:00.967447 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:00.967425 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0906d8be50774c4e8328161c8119d16.slice/crio-0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a WatchSource:0}: Error finding container 0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a: Status 404 returned error can't find the container with id 0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a Oct 30 19:16:00.967971 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:00.967955 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb181debb656c62bb70f1078166094d.slice/crio-26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557 WatchSource:0}: Error finding container 26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557: Status 404 returned error can't find the container with id 26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557 Oct 30 19:16:00.973060 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:00.973039 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:16:01.058947 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.058901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p74tq" Oct 30 19:16:01.067168 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.067143 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6cc8cd_4e1e_472a_97a1_75b4bf3f01ab.slice/crio-618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a WatchSource:0}: Error finding container 618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a: Status 404 returned error can't find the container with id 618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a Oct 30 19:16:01.101621 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.101605 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:16:01.102723 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.102702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:16:01.109938 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.109819 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef62b57_79b3_4002_ba56_d41d3b74ded9.slice/crio-b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73 WatchSource:0}: Error finding container b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73: Status 404 returned error can't find the container with id b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73 Oct 30 19:16:01.111937 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.111918 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f48b9b_ec52_44f6_a537_e6e585762c7e.slice/crio-7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7 WatchSource:0}: Error finding container 7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7: Status 404 returned error can't find the container with id 7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7 Oct 30 19:16:01.116105 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.116088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:16:01.124597 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.124574 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3489239b_c17e_41ac_84f3_392b94d019e5.slice/crio-1bf6c52e9c93a02c0998285e7d779e6a4426d2e55e79c73d864dc0e2052747a3 WatchSource:0}: Error finding container 1bf6c52e9c93a02c0998285e7d779e6a4426d2e55e79c73d864dc0e2052747a3: Status 404 returned error can't find the container with id 1bf6c52e9c93a02c0998285e7d779e6a4426d2e55e79c73d864dc0e2052747a3 Oct 30 19:16:01.131069 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.131055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:16:01.138145 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.138120 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15140033_be3b_4729_9a78_3a38b3d1ef48.slice/crio-227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c WatchSource:0}: Error finding container 227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c: Status 404 returned error can't find the container with id 227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c Oct 30 19:16:01.171637 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.171621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:01.174778 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.174761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:01.180957 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.180936 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79323c8f_776a_4a3e_942c_a467e8672c53.slice/crio-41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa WatchSource:0}: Error finding container 41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa: Status 404 returned error can't find the container with id 41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa Oct 30 19:16:01.182876 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.182843 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bc4ca2_8079_4560_8891_dc52f796c72f.slice/crio-a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22 WatchSource:0}: Error finding container a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22: Status 404 returned error can't find the container with id a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22 Oct 30 19:16:01.189061 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.189046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:16:01.195616 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.195604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w2lmc" Oct 30 19:16:01.197541 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.197523 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df29a6a_001e_463a_9a07_0cfe1c0fe6a0.slice/crio-166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa WatchSource:0}: Error finding container 166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa: Status 404 returned error can't find the container with id 166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa Oct 30 19:16:01.203468 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:01.203450 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862b83b6_5666_4f55_a276_4cb674c0b4a8.slice/crio-06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3 WatchSource:0}: Error finding container 06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3: Status 404 returned error can't find the container with id 06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3 Oct 30 19:16:01.442841 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.442789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:01.442980 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.442843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:01.442980 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.442955 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:01.442980 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.442956 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:01.443215 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.443007 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:02.442991169 +0000 UTC m=+3.069896177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:01.443215 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.443027 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:02.443015693 +0000 UTC m=+3.069920705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:01.529923 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.529140 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:16:01.544176 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.544145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:01.544424 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.544404 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:01.544522 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.544430 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:01.544522 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.544442 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:01.544522 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.544514 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:02.54448263 +0000 UTC m=+3.171387645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:01.771020 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.770908 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:11:00 +0000 UTC" deadline="2028-06-03 06:23:59.876063902 +0000 UTC" Oct 30 19:16:01.771020 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.770975 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13955h7m58.105093438s" Oct 30 19:16:01.883096 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.883049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:01.883258 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:01.883196 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:01.893713 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.893672 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9ssdd" event={"ID":"26bc4ca2-8079-4560-8891-dc52f796c72f","Type":"ContainerStarted","Data":"a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22"} Oct 30 19:16:01.897446 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.897364 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mwhv4" event={"ID":"15140033-be3b-4729-9a78-3a38b3d1ef48","Type":"ContainerStarted","Data":"227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c"} Oct 30 19:16:01.900166 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.900111 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ddx75" event={"ID":"3489239b-c17e-41ac-84f3-392b94d019e5","Type":"ContainerStarted","Data":"1bf6c52e9c93a02c0998285e7d779e6a4426d2e55e79c73d864dc0e2052747a3"} Oct 30 19:16:01.918279 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.918216 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" event={"ID":"53f48b9b-ec52-44f6-a537-e6e585762c7e","Type":"ContainerStarted","Data":"7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7"} Oct 30 19:16:01.925854 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.925785 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerStarted","Data":"b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73"} Oct 30 19:16:01.931574 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.931499 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p74tq" event={"ID":"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab","Type":"ContainerStarted","Data":"618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a"} Oct 30 19:16:01.937511 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.937404 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557"} Oct 30 19:16:01.940027 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.939956 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w2lmc" event={"ID":"862b83b6-5666-4f55-a276-4cb674c0b4a8","Type":"ContainerStarted","Data":"06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3"} Oct 30 19:16:01.941559 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.941537 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkrdd" event={"ID":"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0","Type":"ContainerStarted","Data":"166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa"} Oct 30 19:16:01.943326 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.943268 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa"} Oct 30 19:16:01.953699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:01.953662 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a"} Oct 30 19:16:02.453483 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.453417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:02.453483 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.453475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:02.453711 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.453665 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:02.453762 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.453726 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:04.453707684 +0000 UTC m=+5.080612689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:02.454230 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.454210 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:02.454327 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.454275 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:04.454260779 +0000 UTC m=+5.081165785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:02.554823 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.554794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:02.554987 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.554943 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:02.554987 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.554956 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:02.554987 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.554964 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:02.555171 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.555000 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:04.554989827 +0000 UTC m=+5.181894827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:02.615440 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.615368 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:16:02.771234 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.771121 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:11:00 +0000 UTC" deadline="2028-07-18 17:22:07.250582606 +0000 UTC" Oct 30 19:16:02.771234 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.771190 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15046h6m4.479398079s" Oct 30 19:16:02.872733 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.872701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:02.872895 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.872825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:02.873251 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:02.873226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:02.873377 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:02.873355 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:03.477072 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:03.477044 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:16:03.873266 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:03.872845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:03.873266 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:03.872989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:04.471940 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:04.471853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:04.471940 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:04.471926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:04.472143 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.472044 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:04.472143 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.472098 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:08.472082055 +0000 UTC m=+9.098987071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:04.472257 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.472160 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:04.472257 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.472196 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:08.472184776 +0000 UTC m=+9.099089792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:04.572792 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:04.572764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:04.572969 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.572942 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:04.572969 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.572962 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:04.573076 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.572973 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:04.573076 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.573029 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:08.57300977 +0000 UTC m=+9.199914785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:04.873139 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:04.872395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:04.873139 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.872515 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:04.873139 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:04.872890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:04.873139 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:04.873001 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:05.872774 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:05.872691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:05.872964 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:05.872788 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:05.987510 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:05.987429 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"0bda5fbf6e9e2678ca515373c50ec8ae99e8355eaf78dc416d893d716a40523b"} Oct 30 19:16:05.989937 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:05.989913 2579 generic.go:356] "Generic (PLEG): container finished" podID="b0906d8be50774c4e8328161c8119d16" containerID="01f2d98a242cdce7216d71858868589efc34721cead3d7075f93edd5386d5205" exitCode=0 Oct 30 19:16:05.990049 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:05.989985 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerDied","Data":"01f2d98a242cdce7216d71858868589efc34721cead3d7075f93edd5386d5205"} Oct 30 19:16:05.996978 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:05.996955 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9ssdd" event={"ID":"26bc4ca2-8079-4560-8891-dc52f796c72f","Type":"ContainerStarted","Data":"a37b1e8d05a49af810b6a6251440fa0fd5bae70a7cff7c5a4de60476513b4111"} Oct 30 19:16:06.002891 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:06.002014 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" podStartSLOduration=6.002000152 podStartE2EDuration="6.002000152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:06.002001485 +0000 UTC m=+6.628906513" watchObservedRunningTime="2026-10-30 19:16:06.002000152 +0000 UTC m=+6.628905174" Oct 30 19:16:06.003033 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:06.003012 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="52aaa5f8574c158d5763046b1a8a6d64f35f35eab0b739e47c87ba677a6fcf50" exitCode=0 Oct 30 19:16:06.003126 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:06.003041 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerDied","Data":"52aaa5f8574c158d5763046b1a8a6d64f35f35eab0b739e47c87ba677a6fcf50"} Oct 30 19:16:06.016406 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:06.015434 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9ssdd" podStartSLOduration=2.735544521 podStartE2EDuration="7.015418655s" totalImagesPullingTime="4.279874134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:06.015349535 +0000 UTC m=+6.642254559" watchObservedRunningTime="2026-10-30 19:16:06.015418655 +0000 UTC m=+6.642323680" Oct 30 19:16:06.873234 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:06.873199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:06.873390 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:06.873337 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:06.876056 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:06.876027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:06.876230 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:06.876203 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:07.013474 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:07.013442 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkrdd" event={"ID":"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0","Type":"ContainerStarted","Data":"42a5440547ee65b0bed44925cc2e531f369fd43dbe0f3e86abb391fc39f9745d"} Oct 30 19:16:07.018597 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:07.017412 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"e89b039a1f76e192410796bf6b847693a250491066b525cffab0c527ee63cb56"} Oct 30 19:16:07.029450 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:07.029411 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mkrdd" podStartSLOduration=1.486370167 podStartE2EDuration="7.029398396s" totalImagesPullingTime="5.543028229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:07.029205825 +0000 UTC m=+7.656110843" watchObservedRunningTime="2026-10-30 19:16:07.029398396 +0000 UTC m=+7.656303418" Oct 30 19:16:07.044087 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:07.044044 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" podStartSLOduration=7.044028403 podStartE2EDuration="7.044028403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:07.043659599 +0000 UTC m=+7.670564621" watchObservedRunningTime="2026-10-30 19:16:07.044028403 +0000 UTC m=+7.670933427" Oct 30 19:16:07.873230 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:07.872841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:07.873230 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:07.872972 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:08.023066 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.023034 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" event={"ID":"53f48b9b-ec52-44f6-a537-e6e585762c7e","Type":"ContainerStarted","Data":"f8e41ce931321a3b86f5fa217aee0f88b76386453d948efa01656cd4f51a68ce"} Oct 30 19:16:08.506672 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.506637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:08.506848 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.506725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:08.506943 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.506847 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:08.506943 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.506918 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.506901241 +0000 UTC m=+17.133806247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:08.507188 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.507149 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:08.507272 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.507256 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.507238061 +0000 UTC m=+17.134143064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:08.517178 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.517147 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:08.608147 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.608103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:08.608600 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.608578 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:08.608680 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.608606 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:08.608680 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.608621 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:08.608680 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.608670 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.60865264 +0000 UTC m=+17.235557653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:08.768651 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.768598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.829254 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.829228 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:16:08.873381 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.873352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:08.873512 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.873486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:08.873579 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:08.873568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:08.873677 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:08.873656 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:09.026185 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:09.026036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:09.029456 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:09.029434 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" event={"ID":"53f48b9b-ec52-44f6-a537-e6e585762c7e","Type":"ContainerStarted","Data":"735b3ffe5ab2c536fc38d99d76982f230341d3485d7801afcdfa88dbfd0d9999"} Oct 30 19:16:09.795155 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:09.794944 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:16:08.829250539Z","UUID":"232eb09e-1ae8-42bb-8c69-afb8c70c0aba","Handler":null,"Name":"","Endpoint":""} Oct 30 19:16:09.797375 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:09.796778 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:16:09.797375 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:09.796819 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:16:09.877772 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:09.877742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:09.877984 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:09.877888 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:10.033761 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.033556 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w2lmc" event={"ID":"862b83b6-5666-4f55-a276-4cb674c0b4a8","Type":"ContainerStarted","Data":"db7a1aa0d835dd0f0cb3d10ce1e3d659319ff9995cae8051c845e8ccc0b636dc"} Oct 30 19:16:10.036518 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.036492 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p74tq" event={"ID":"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab","Type":"ContainerStarted","Data":"2528ddf4a775e24e428582633890efcec21876096f743789a5bac88ea09e2300"} Oct 30 19:16:10.072495 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.072455 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w2lmc" podStartSLOduration=2.284899077 podStartE2EDuration="10.072442741s" totalImagesPullingTime="7.787543664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:10.053439355 +0000 UTC m=+10.680344377" watchObservedRunningTime="2026-10-30 19:16:10.072442741 +0000 UTC m=+10.699347763" Oct 30 19:16:10.072808 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.072788 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p74tq" podStartSLOduration=2.786970235 podStartE2EDuration="11.072783551s" totalImagesPullingTime="8.285813316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:10.07214583 +0000 UTC m=+10.699050852" watchObservedRunningTime="2026-10-30 19:16:10.072783551 +0000 UTC m=+10.699688573" Oct 30 19:16:10.429169 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.429097 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:10.429586 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.429567 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:10.872714 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.872612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:10.872714 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:10.872613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:10.873029 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:10.872737 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:10.873029 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:10.872820 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:11.042564 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.042058 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" event={"ID":"53f48b9b-ec52-44f6-a537-e6e585762c7e","Type":"ContainerStarted","Data":"fe4c2a21f4afc83a10f10ac50b425d9cf2e55404e7509ce3ca10bfa000136ad9"} Oct 30 19:16:11.044675 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.044649 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="eca04dd31615fb71f75f7561c8a61f21574250ee015ae77a10665d82958e8551" exitCode=0 Oct 30 19:16:11.044889 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.044851 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerDied","Data":"eca04dd31615fb71f75f7561c8a61f21574250ee015ae77a10665d82958e8551"} Oct 30 19:16:11.045807 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.045762 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:16:11.060676 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.060630 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" podStartSLOduration=3.596728609 podStartE2EDuration="12.060613873s" totalImagesPullingTime="8.463885264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:11.059198744 +0000 UTC m=+11.686103768" watchObservedRunningTime="2026-10-30 19:16:11.060613873 +0000 UTC m=+11.687518896" Oct 30 19:16:11.183356 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.183289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:11.872890 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:11.872854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:11.872997 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:11.872975 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:12.049161 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.049076 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="3dd302ad56ebd9720f8eed8b6a3db81ab2b21bab37c4d0553ec4ed2a6e24e9bf" exitCode=0 Oct 30 19:16:12.049646 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.049161 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerDied","Data":"3dd302ad56ebd9720f8eed8b6a3db81ab2b21bab37c4d0553ec4ed2a6e24e9bf"} Oct 30 19:16:12.051601 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.051572 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mwhv4" event={"ID":"15140033-be3b-4729-9a78-3a38b3d1ef48","Type":"ContainerStarted","Data":"1a337a561c23dd6617e77b90e44b12f2fa208fbfda9650e3f3a9c036555560d2"} Oct 30 19:16:12.053439 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.053403 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ddx75" event={"ID":"3489239b-c17e-41ac-84f3-392b94d019e5","Type":"ContainerStarted","Data":"5d5fd0e5d1c649b0de2b6c32a5f616f48578b983e3c42ec2a6858892772b3b07"} Oct 30 19:16:12.086641 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.086593 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ddx75" podStartSLOduration=2.925094404 podStartE2EDuration="13.08657941s" totalImagesPullingTime="10.161485006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:12.086158087 +0000 UTC m=+12.713063109" watchObservedRunningTime="2026-10-30 19:16:12.08657941 +0000 UTC m=+12.713484431" Oct 30 19:16:12.090360 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.090335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:12.101404 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.101365 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mwhv4" podStartSLOduration=4.708700476 podStartE2EDuration="13.10135125s" totalImagesPullingTime="8.392650774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:12.100983306 +0000 UTC m=+12.727888328" watchObservedRunningTime="2026-10-30 19:16:12.10135125 +0000 UTC m=+12.728256273" Oct 30 19:16:12.872773 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.872739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:12.872937 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:12.872851 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:12.872937 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:12.872904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:12.873044 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:12.873012 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:13.148855 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:13.148830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:13.257111 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:13.257085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:13.873345 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:13.873318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:13.873524 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:13.873425 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:14.060045 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:14.060014 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="b21a68be9b27e2a3e1818ddd35952d4bea2ac5cb7734e58ae8cc06626948d3d8" exitCode=0 Oct 30 19:16:14.060205 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:14.060073 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerDied","Data":"b21a68be9b27e2a3e1818ddd35952d4bea2ac5cb7734e58ae8cc06626948d3d8"} Oct 30 19:16:14.873290 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:14.873218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:14.873891 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:14.873334 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:14.873891 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:14.873602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:14.873891 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:14.873685 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:15.200638 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.200384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.306440 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.306401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.406630 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.406603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.532941 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.532917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.637442 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.637421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.744827 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.744776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.873157 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:15.872962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:15.873275 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:15.873249 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:16.035934 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.035889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:16.066833 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.066811 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="76cef0201b34454b210a1ec92af1a7e2283e46adc0395a65e7cad5e0365d01cc" exitCode=0 Oct 30 19:16:16.066925 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.066898 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerDied","Data":"76cef0201b34454b210a1ec92af1a7e2283e46adc0395a65e7cad5e0365d01cc"} Oct 30 19:16:16.069546 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.069532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:16:16.069841 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.069825 2579 generic.go:356] "Generic (PLEG): container finished" podID="79323c8f-776a-4a3e-942c-a467e8672c53" containerID="45872986aa95c74662be8ea04a9ddf2b175c5992556ac9b5a77deb536bc845e1" exitCode=1 Oct 30 19:16:16.069915 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.069872 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"b0781f81585199a20a1e23b41215918a5565810dad8e8c3045f3503fec4dfcf5"} Oct 30 19:16:16.069948 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.069923 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"d33b3edaf3bd0379ca6a3328a59f3d99b106b1b47f3a5034d4d370a71ec802c6"} Oct 30 19:16:16.069976 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.069951 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"5c7f0e3ad1fb3c5f7f217b76e2bbc099b07b194fb8f7a999fbe149fc6549f259"} Oct 30 19:16:16.070006 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.069978 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"de63f13d36d12441159893b12c4804b6e80abbc62c07c6620693cb1d762ef874"} Oct 30 19:16:16.070006 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.070004 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerDied","Data":"45872986aa95c74662be8ea04a9ddf2b175c5992556ac9b5a77deb536bc845e1"} Oct 30 19:16:16.070060 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.070032 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"1419bebdc2dd0bb22ca9b6565c7e26269ee2541a004e409cb02f3a6522b55a47"} Oct 30 19:16:16.144732 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.144711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:16.570731 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.570707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:16.570854 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.570742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:16.570907 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.570850 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:16.570941 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.570913 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:32.570900739 +0000 UTC m=+33.197805752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:16.570988 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.570850 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:16.570988 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.570981 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:32.570970343 +0000 UTC m=+33.197875358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:16.671273 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.671249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:16.671385 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.671355 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:16.671385 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.671376 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:16.671489 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.671387 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:16.671489 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.671436 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:32.671421308 +0000 UTC m=+33.298326323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:16.702244 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.702223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:16.873042 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.873003 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:16.873042 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:16.873029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:16.873143 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.873104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:16.873241 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:16.873219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:17.074450 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:17.074430 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="4d6ed7bff4e7d31f7a6eebb5e39f29c351337fe4f6e523553a3a14781fbab91d" exitCode=0 Oct 30 19:16:17.074813 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:17.074475 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerDied","Data":"4d6ed7bff4e7d31f7a6eebb5e39f29c351337fe4f6e523553a3a14781fbab91d"} Oct 30 19:16:17.753151 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:17.753129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:17.860630 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:17.860605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:17.872714 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:17.872698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:17.872785 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:17.872772 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:17.965114 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:17.965091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.080176 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:18.080119 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" event={"ID":"9ef62b57-79b3-4002-ba56-d41d3b74ded9","Type":"ContainerStarted","Data":"5c682f5bd160885beebaf9d470b03a0be89ca0b729c42e367d3e7b183bb2b78d"} Oct 30 19:16:18.102844 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:18.102806 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xbzbq" podStartSLOduration=8.632014831 podStartE2EDuration="19.102795676s" totalImagesPullingTime="10.470780845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:18.101264482 +0000 UTC m=+18.728169503" watchObservedRunningTime="2026-10-30 19:16:18.102795676 +0000 UTC m=+18.729700697" Oct 30 19:16:18.189932 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:18.189908 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p74tq_1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab/dns-node-resolver/0.log" Oct 30 19:16:18.204534 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:18.204503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.872826 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:18.872804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:18.872947 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:18.872929 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:18.873011 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:18.872985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:18.873105 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:18.873089 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:19.084452 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.084429 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:16:19.085397 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.085368 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"b295a2c6d9781e65bca8a8c40f5af2996837c6d91c98ee1aca3668540933806f"} Oct 30 19:16:19.189991 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.189935 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkrdd_1df29a6a-001e-463a-9a07-0cfe1c0fe6a0/node-ca/0.log" Oct 30 19:16:19.404202 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.404173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:19.602204 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.602142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:19.799164 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.799143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:19.873510 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.873459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:19.873601 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:19.873524 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:19.904951 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:19.904929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:20.018948 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:20.018924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:20.203781 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:20.203749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:20.304979 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:20.304752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:20.410379 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:20.410346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:20.872934 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:20.872911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:20.873054 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:20.873005 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:20.873088 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:20.873060 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:20.873155 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:20.873139 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:21.091675 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.091654 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:16:21.092069 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.092047 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"709d3d5ad94fd184144b606dbbc4c1d9f066716e83020b466f06cf889ff05d76"} Oct 30 19:16:21.092308 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.092288 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:21.092430 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.092366 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:21.092504 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.092489 2579 scope.go:122] "RemoveContainer" containerID="45872986aa95c74662be8ea04a9ddf2b175c5992556ac9b5a77deb536bc845e1" Oct 30 19:16:21.108190 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.108171 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:21.401226 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.401178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:21.673767 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.673723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:21.817167 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.817142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:21.873387 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:21.873365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:21.873486 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:21.873467 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:22.000954 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.000908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:22.097592 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.097574 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:16:22.097914 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.097895 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" event={"ID":"79323c8f-776a-4a3e-942c-a467e8672c53","Type":"ContainerStarted","Data":"9c1a14e218ccd261f13d79f8409ebbe1b770882782c412f3aa6f804cd050899a"} Oct 30 19:16:22.098098 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.098084 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:22.113802 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.113777 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:22.126230 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.126195 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" podStartSLOduration=8.834680284000001 podStartE2EDuration="23.126185642s" totalImagesPullingTime="14.291505358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:22.124488962 +0000 UTC m=+22.751393984" watchObservedRunningTime="2026-10-30 19:16:22.126185642 +0000 UTC m=+22.753090663" Oct 30 19:16:22.206360 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.206340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:22.312241 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.312191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:22.873024 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.873003 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:22.873395 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:22.873004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:22.873395 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:22.873114 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:22.873395 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:22.873160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:23.401232 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:23.401210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.606909 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:23.606889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.813667 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:23.813621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.872307 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:23.872290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:23.872416 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:23.872396 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:23.999829 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:23.999804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:24.202621 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:24.202569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:24.354262 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:24.354242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:24.872890 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:24.872849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:24.873006 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:24.872853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:24.873056 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:24.873027 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:24.873056 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:24.872951 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:25.414017 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:25.413976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.538567 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:25.538538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.641085 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:25.641064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.805904 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:25.805829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.873314 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:25.873291 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:25.873404 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:25.873375 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:26.008755 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:26.008732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:26.201550 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:26.201490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:26.496487 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:26.496432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:26.872767 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:26.872722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:26.872853 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:26.872806 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:26.872920 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:26.872873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:26.872968 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:26.872953 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:27.198990 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:27.198913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:27.311012 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:27.310983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:27.626459 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:27.626403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:27.854193 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:27.854168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:27.873219 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:27.873203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:27.873308 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:27.873292 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:27.986650 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:27.986602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.131972 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:28.131953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.295422 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:28.295354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.615548 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:28.615490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.872734 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:28.872685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:28.872734 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:28.872708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:28.873080 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:28.872779 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:28.873080 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:28.872893 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:29.041198 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:29.041171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:29.874069 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:29.874041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:29.874400 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:29.874140 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:30.873155 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:30.873127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:30.873300 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:30.873227 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:30.873366 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:30.873312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:30.873421 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:30.873400 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:31.872411 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:31.872383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:31.872723 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:31.872490 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:32.584043 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.584016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:32.584241 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.584050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:32.584241 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.584156 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:32.584241 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.584194 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:32.584241 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.584210 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret podName:6b654842-b7a7-4e8d-8c92-8729111d0373 nodeName:}" failed. No retries permitted until 2026-10-30 19:17:04.584193755 +0000 UTC m=+65.211098757 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret") pod "global-pull-secret-syncer-5sqtd" (UID: "6b654842-b7a7-4e8d-8c92-8729111d0373") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:32.584241 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.584228 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs podName:06348567-413f-417b-a78b-a10d7ff83ff4 nodeName:}" failed. No retries permitted until 2026-10-30 19:17:04.584217395 +0000 UTC m=+65.211122396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs") pod "network-metrics-daemon-lc88s" (UID: "06348567-413f-417b-a78b-a10d7ff83ff4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:32.678453 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.678432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:32.685061 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.685044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:32.685163 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.685151 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:32.685197 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.685166 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:32.685197 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.685175 2579 projected.go:196] Error preparing data for projected volume kube-api-access-th98d for pod openshift-network-diagnostics/network-check-target-tfj5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:32.685253 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.685218 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d podName:e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f nodeName:}" failed. No retries permitted until 2026-10-30 19:17:04.685206726 +0000 UTC m=+65.312111739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-th98d" (UniqueName: "kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d") pod "network-check-target-tfj5g" (UID: "e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:32.778359 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.778339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:32.872935 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.872892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:32.873199 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.872983 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:32.873199 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.873009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:32.873199 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:32.873081 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:32.884474 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.884450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:32.986673 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:32.986653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.099329 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:33.099310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.213342 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:33.213297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.569463 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:33.569412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.665266 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:33.665237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.873198 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:33.873161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:33.873481 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:33.873245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:34.873197 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:34.873174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:34.873498 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:34.873174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:34.873498 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:34.873264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:34.873498 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:34.873343 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:35.872643 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:35.872619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:35.872765 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:35.872707 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:36.473168 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:36.473137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:36.604844 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:36.604821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:36.873357 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:36.873295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:36.873485 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:36.873305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:36.873485 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:36.873376 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:36.873485 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:36.873464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:37.872920 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:37.872755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:37.873342 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:37.873042 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:38.010001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.009161 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tfj5g"] Oct 30 19:16:38.010001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.009281 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc88s"] Oct 30 19:16:38.010001 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.009412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:38.010001 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:38.009574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:38.011519 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.011491 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5sqtd"] Oct 30 19:16:38.011641 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.011584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:38.011706 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:38.011665 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:38.127663 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.127641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:38.127778 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:38.127738 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:38.799006 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:38.798951 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p74tq_1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab/dns-node-resolver/0.log" Oct 30 19:16:39.168424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.168360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:39.290349 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.290317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:39.401917 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.401891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:39.874062 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.874032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:39.874228 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:39.874121 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:39.874228 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.874196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:39.874321 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:39.874285 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:39.874369 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.874360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:39.874459 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:39.874442 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:39.998075 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:39.998053 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkrdd_1df29a6a-001e-463a-9a07-0cfe1c0fe6a0/node-ca/0.log" Oct 30 19:16:41.378699 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:41.378613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:41.873177 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:41.873158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:41.873290 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:41.873186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:41.873290 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:41.873158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:41.873290 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:41.873251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lc88s" podUID="06348567-413f-417b-a78b-a10d7ff83ff4" Oct 30 19:16:41.873381 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:41.873328 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfj5g" podUID="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" Oct 30 19:16:41.873416 ip-10-0-138-233 kubenswrapper[2579]: E1030 19:16:41.873400 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sqtd" podUID="6b654842-b7a7-4e8d-8c92-8729111d0373" Oct 30 19:16:42.621838 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.621777 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeReady" Oct 30 19:16:42.622367 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.621911 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:16:42.668037 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.668009 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fjmnw"] Oct 30 19:16:42.670058 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.670043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.671586 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.671566 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-54p6s"] Oct 30 19:16:42.673248 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.673233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.676180 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.676154 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:16:42.676349 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.676330 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:16:42.676409 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.676374 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfmhh\"" Oct 30 19:16:42.676913 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.676894 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:16:42.677009 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.676956 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:16:42.677106 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.677091 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:16:42.680402 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.680384 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7b7kv\"" Oct 30 19:16:42.681598 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.681579 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fjmnw"] Oct 30 19:16:42.688831 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.688813 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54p6s"] Oct 30 19:16:42.696144 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.696124 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bpk2j"] Oct 30 19:16:42.697770 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.697754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.712218 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.712195 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bpk2j"] Oct 30 19:16:42.752281 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c346bfd4-9367-4a93-a44e-022b444c9f8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.752373 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvlc\" (UniqueName: \"kubernetes.io/projected/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-api-access-chvlc\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.752373 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-metrics-tls\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.752488 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c346bfd4-9367-4a93-a44e-022b444c9f8c-data-volume\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.752488 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d5763-db1f-46f7-99fe-d55f45fa9a76-cert\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.752488 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c346bfd4-9367-4a93-a44e-022b444c9f8c-crio-socket\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.752637 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.752637 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcq4\" (UniqueName: \"kubernetes.io/projected/471d5763-db1f-46f7-99fe-d55f45fa9a76-kube-api-access-tlcq4\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.752637 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-config-volume\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.752637 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-tmp-dir\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.752637 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.752606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2d7j\" (UniqueName: \"kubernetes.io/projected/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-kube-api-access-m2d7j\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.776253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.776228 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:16:42.776253 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.776246 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:16:42.776407 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.776228 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bfjrl\"" Oct 30 19:16:42.776407 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.776300 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:16:42.776407 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.776344 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:16:42.853072 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c346bfd4-9367-4a93-a44e-022b444c9f8c-crio-socket\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853169 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853169 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcq4\" (UniqueName: \"kubernetes.io/projected/471d5763-db1f-46f7-99fe-d55f45fa9a76-kube-api-access-tlcq4\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.853169 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-config-volume\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.853302 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-tmp-dir\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.853302 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c346bfd4-9367-4a93-a44e-022b444c9f8c-crio-socket\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853493 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-tmp-dir\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.853552 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2d7j\" (UniqueName: \"kubernetes.io/projected/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-kube-api-access-m2d7j\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.853604 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c346bfd4-9367-4a93-a44e-022b444c9f8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853604 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvlc\" (UniqueName: \"kubernetes.io/projected/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-api-access-chvlc\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853706 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-metrics-tls\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.853706 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c346bfd4-9367-4a93-a44e-022b444c9f8c-data-volume\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853706 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d5763-db1f-46f7-99fe-d55f45fa9a76-cert\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.853840 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.853840 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.853818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-config-volume\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.854040 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.854020 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c346bfd4-9367-4a93-a44e-022b444c9f8c-data-volume\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.857958 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.857937 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-metrics-tls\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.858026 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.857940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c346bfd4-9367-4a93-a44e-022b444c9f8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.858068 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.858037 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d5763-db1f-46f7-99fe-d55f45fa9a76-cert\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.862207 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.862161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2d7j\" (UniqueName: \"kubernetes.io/projected/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-kube-api-access-m2d7j\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.862207 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.862164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvlc\" (UniqueName: \"kubernetes.io/projected/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-api-access-chvlc\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:42.862789 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.862767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcq4\" (UniqueName: \"kubernetes.io/projected/471d5763-db1f-46f7-99fe-d55f45fa9a76-kube-api-access-tlcq4\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:42.983815 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.983789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:42.992373 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:42.992355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:16:43.007118 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.007100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:16:43.159841 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.159667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.282467 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.281195 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fjmnw"] Oct 30 19:16:43.282467 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.282027 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54p6s"] Oct 30 19:16:43.285526 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:43.285501 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9c1b81_6bc0_406b_a0c1_a370afa909b0.slice/crio-1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266 WatchSource:0}: Error finding container 1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266: Status 404 returned error can't find the container with id 1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266 Oct 30 19:16:43.286162 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:43.286141 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc346bfd4_9367_4a93_a44e_022b444c9f8c.slice/crio-e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526 WatchSource:0}: Error finding container e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526: Status 404 returned error can't find the container with id e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526 Oct 30 19:16:43.297815 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.297797 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bpk2j"] Oct 30 19:16:43.298712 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.298681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.301156 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:43.301136 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471d5763_db1f_46f7_99fe_d55f45fa9a76.slice/crio-0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b WatchSource:0}: Error finding container 0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b: Status 404 returned error can't find the container with id 0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b Oct 30 19:16:43.416142 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.416113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.543572 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.543525 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz"] Oct 30 19:16:43.545513 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.545497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.550818 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.550788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.551703 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.551443 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:16:43.551776 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.551747 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:16:43.551776 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.551767 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:16:43.551901 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.551801 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:16:43.551901 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.551826 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:16:43.560948 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.560929 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz"] Oct 30 19:16:43.579272 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.579249 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ftfsr"] Oct 30 19:16:43.581291 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.581278 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf"] Oct 30 19:16:43.581414 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.581399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.583281 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.583266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.594800 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.594781 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf"] Oct 30 19:16:43.603173 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603154 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:16:43.603248 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603178 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bc6qp\"" Oct 30 19:16:43.603423 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603408 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:16:43.603532 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603522 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:16:43.603657 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603645 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:16:43.603689 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603649 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jbwq9\"" Oct 30 19:16:43.603737 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.603724 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jzpj7\"" Oct 30 19:16:43.649984 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.649962 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:16:43.650381 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.649980 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:16:43.660258 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-metrics-client-ca\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660333 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-wtmp\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660333 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclx4\" (UniqueName: \"kubernetes.io/projected/faee8bf1-c317-4506-a5a7-e7317ab3909d-kube-api-access-mclx4\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660333 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-tls\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660430 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660430 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fb8c089-70f4-418d-8cd4-fd2393997ccf-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.660498 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.660498 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da476d5e-0b9c-428d-a53a-98f99d2b15c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.660557 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.660557 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stb86\" (UniqueName: \"kubernetes.io/projected/da476d5e-0b9c-428d-a53a-98f99d2b15c5-kube-api-access-stb86\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.660619 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.660619 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.660619 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-sys\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660715 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.660715 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g88x\" (UniqueName: \"kubernetes.io/projected/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-api-access-4g88x\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.660781 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-root\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660781 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.660781 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.660940 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.660793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-textfile\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.678149 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.678127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.761976 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.761945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-root\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762088 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.761979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.762088 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762088 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-textfile\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762088 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-metrics-client-ca\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762278 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-wtmp\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762278 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-root\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762278 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762238 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-wtmp\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762412 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclx4\" (UniqueName: \"kubernetes.io/projected/faee8bf1-c317-4506-a5a7-e7317ab3909d-kube-api-access-mclx4\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762412 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-tls\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762412 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-textfile\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762412 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762610 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fb8c089-70f4-418d-8cd4-fd2393997ccf-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.762610 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.762610 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da476d5e-0b9c-428d-a53a-98f99d2b15c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.762745 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.762745 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stb86\" (UniqueName: \"kubernetes.io/projected/da476d5e-0b9c-428d-a53a-98f99d2b15c5-kube-api-access-stb86\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.762745 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.762745 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.762745 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.762745 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-sys\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.763079 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.763079 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-metrics-client-ca\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.763079 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.763079 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.762920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g88x\" (UniqueName: \"kubernetes.io/projected/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-api-access-4g88x\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.763268 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.763090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-sys\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.763392 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.763367 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da476d5e-0b9c-428d-a53a-98f99d2b15c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.763593 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.763573 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fb8c089-70f4-418d-8cd4-fd2393997ccf-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.763907 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.763709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.766572 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.766554 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.766662 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.766641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.768540 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.768514 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.768738 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.768555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.768831 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.768791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-tls\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.770014 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.769996 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.770474 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.770451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclx4\" (UniqueName: \"kubernetes.io/projected/faee8bf1-c317-4506-a5a7-e7317ab3909d-kube-api-access-mclx4\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.771220 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.771202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stb86\" (UniqueName: \"kubernetes.io/projected/da476d5e-0b9c-428d-a53a-98f99d2b15c5-kube-api-access-stb86\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.771489 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.771473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g88x\" (UniqueName: \"kubernetes.io/projected/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-api-access-4g88x\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.778491 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.778469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.857766 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.857704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:16:43.874853 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.874829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:16:43.875233 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.875216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:16:43.875625 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.875600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.880525 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.880589 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.880640 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\"" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.880805 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.880835 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\"" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.880855 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:16:43.882363 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.881602 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\"" Oct 30 19:16:43.883510 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.883385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.893254 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.893237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:16:43.898455 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.898434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:16:43.907857 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:43.907828 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaee8bf1_c317_4506_a5a7_e7317ab3909d.slice/crio-ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3 WatchSource:0}: Error finding container ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3: Status 404 returned error can't find the container with id ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3 Oct 30 19:16:43.994438 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:43.994337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.106612 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.106535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.145678 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.145652 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz"] Oct 30 19:16:44.154791 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.154757 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54p6s" event={"ID":"c346bfd4-9367-4a93-a44e-022b444c9f8c","Type":"ContainerStarted","Data":"e53cb3de497ee46c9b961802bfd127ef97774c9d1c3e89ae0c7281206f6e95f3"} Oct 30 19:16:44.154910 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.154824 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54p6s" event={"ID":"c346bfd4-9367-4a93-a44e-022b444c9f8c","Type":"ContainerStarted","Data":"e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526"} Oct 30 19:16:44.156940 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.156912 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fjmnw" event={"ID":"3c9c1b81-6bc0-406b-a0c1-a370afa909b0","Type":"ContainerStarted","Data":"1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266"} Oct 30 19:16:44.159943 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.159917 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftfsr" event={"ID":"faee8bf1-c317-4506-a5a7-e7317ab3909d","Type":"ContainerStarted","Data":"ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3"} Oct 30 19:16:44.161456 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.161431 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bpk2j" event={"ID":"471d5763-db1f-46f7-99fe-d55f45fa9a76","Type":"ContainerStarted","Data":"0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b"} Oct 30 19:16:44.205414 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.205383 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf"] Oct 30 19:16:44.210125 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:44.210096 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb8c089_70f4_418d_8cd4_fd2393997ccf.slice/crio-b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0 WatchSource:0}: Error finding container b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0: Status 404 returned error can't find the container with id b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0 Oct 30 19:16:44.389325 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.389072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.569320 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.569284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.626065 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.626028 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:16:44.629934 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.628970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.634512 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.634490 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:16:44.634714 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.634671 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:16:44.634809 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.634728 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:16:44.634809 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.634783 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:16:44.635012 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.634892 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hlrws\"" Oct 30 19:16:44.635012 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.634973 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:16:44.635131 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.635036 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:16:44.635131 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.635110 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:16:44.635208 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.635169 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:16:44.635247 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.635223 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:16:44.647115 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.646989 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:16:44.771972 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.771939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.771981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.772437 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.772391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.805823 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.805756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.873757 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.873757 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.873757 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.873996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874029 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.874020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.874400 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.874058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.875366 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.875336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.877311 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.875804 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.877643 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.877592 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.893317 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.893263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.893513 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.893490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.893590 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.893526 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.894353 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.893933 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.894353 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.894252 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.894353 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.894308 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.894694 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.894655 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.894928 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.894911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.895604 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.895581 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.895713 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.895696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:44.921286 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.921229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.953673 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:44.953648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:16:45.167690 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.167610 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54p6s" event={"ID":"c346bfd4-9367-4a93-a44e-022b444c9f8c","Type":"ContainerStarted","Data":"f621ecf3b52ed7f6708ed15f6c731a8ed7cf5717d6ae158887340f7817e89e2f"} Oct 30 19:16:45.170090 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.170062 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" event={"ID":"da476d5e-0b9c-428d-a53a-98f99d2b15c5","Type":"ContainerStarted","Data":"46fd35fda01cbbf88a129fd3a5de7bef29b26c2cb23749ec258b6596ce3079fe"} Oct 30 19:16:45.170208 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.170121 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" event={"ID":"da476d5e-0b9c-428d-a53a-98f99d2b15c5","Type":"ContainerStarted","Data":"e445abc14521faa3a01d44f9c3d4c33a0d20f813803308d0ba010b2446ac3185"} Oct 30 19:16:45.170208 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.170149 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" event={"ID":"da476d5e-0b9c-428d-a53a-98f99d2b15c5","Type":"ContainerStarted","Data":"0c4a8b38ed11b915f3d6f1316cabb3f69aa1adb696d57c5d8e1438c7c7f6c598"} Oct 30 19:16:45.171611 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.171587 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" event={"ID":"6fb8c089-70f4-418d-8cd4-fd2393997ccf","Type":"ContainerStarted","Data":"b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0"} Oct 30 19:16:45.223388 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.222096 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56c4467959-2ccth"] Oct 30 19:16:45.226738 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.226710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.237358 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.237272 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:16:45.237982 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.237364 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238160 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238255 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238352 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n75qq\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238380 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238434 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238573 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:16:45.238905 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.238756 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56c4467959-2ccth"] Oct 30 19:16:45.275846 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.275816 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:16:45.292045 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:45.292017 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73dfeab_ee8f_47c9_b1b6_1209a37deaa5.slice/crio-03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715 WatchSource:0}: Error finding container 03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715: Status 404 returned error can't find the container with id 03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715 Oct 30 19:16:45.293782 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.293741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:45.378621 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.378747 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.378747 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.378747 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.378915 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.378915 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.378915 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.378839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.480423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.481197 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.482424 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.482156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.483532 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.483502 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.484176 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.484138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.486849 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.486829 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.488111 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.488054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.493288 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.493243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.494846 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.494821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:45.552022 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.551970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:45.858950 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:45.858854 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56c4467959-2ccth"] Oct 30 19:16:45.876175 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:45.876126 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45afd0c1_8540_4a90_a7ee_1db0805c9e2c.slice/crio-9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e WatchSource:0}: Error finding container 9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e: Status 404 returned error can't find the container with id 9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e Oct 30 19:16:46.180696 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.180658 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" event={"ID":"da476d5e-0b9c-428d-a53a-98f99d2b15c5","Type":"ContainerStarted","Data":"c0f2c61aeb35b3fa2dcd474b71be1864c3fd4c620ce90b0773516d84fe988de9"} Oct 30 19:16:46.183469 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.182768 2579 generic.go:356] "Generic (PLEG): container finished" podID="faee8bf1-c317-4506-a5a7-e7317ab3909d" containerID="b717c7fd979f0b5a50fa1fc1a329f7c0245d1f13219d50485ab9db76c9044bab" exitCode=0 Oct 30 19:16:46.183469 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.182850 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftfsr" event={"ID":"faee8bf1-c317-4506-a5a7-e7317ab3909d","Type":"ContainerDied","Data":"b717c7fd979f0b5a50fa1fc1a329f7c0245d1f13219d50485ab9db76c9044bab"} Oct 30 19:16:46.187890 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.187719 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c4467959-2ccth" event={"ID":"45afd0c1-8540-4a90-a7ee-1db0805c9e2c","Type":"ContainerStarted","Data":"9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e"} Oct 30 19:16:46.191971 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.191650 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715"} Oct 30 19:16:46.199900 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.199737 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" podStartSLOduration=1.656484472 podStartE2EDuration="3.199719661s" totalImagesPullingTime="1.543235189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:46.198363826 +0000 UTC m=+46.825268849" watchObservedRunningTime="2026-10-30 19:16:46.199719661 +0000 UTC m=+46.826624682" Oct 30 19:16:46.554310 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.554279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:46.659882 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.659804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:46.763997 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:46.763904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:47.200523 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.200485 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54p6s" event={"ID":"c346bfd4-9367-4a93-a44e-022b444c9f8c","Type":"ContainerStarted","Data":"407edd81684b3f1aca73ab79e916dad50a1a336f8886133d48410a3e8c11a4b8"} Oct 30 19:16:47.202615 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.202580 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fjmnw" event={"ID":"3c9c1b81-6bc0-406b-a0c1-a370afa909b0","Type":"ContainerStarted","Data":"6bd49a442c64c5ddc6f649416c2b5a08f8daba9f659f6f53d1607083459d0615"} Oct 30 19:16:47.202726 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.202636 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fjmnw" event={"ID":"3c9c1b81-6bc0-406b-a0c1-a370afa909b0","Type":"ContainerStarted","Data":"852775d43e2584b4f12928ae3fb975f1575ebc115217b4cd1c9980dd0950e08b"} Oct 30 19:16:47.202726 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.202699 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:47.205068 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.205046 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftfsr" event={"ID":"faee8bf1-c317-4506-a5a7-e7317ab3909d","Type":"ContainerStarted","Data":"b34d60a3257aa10c8da8909839edbd18265741e36d97940fe870e5e62cccca08"} Oct 30 19:16:47.205155 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.205089 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftfsr" event={"ID":"faee8bf1-c317-4506-a5a7-e7317ab3909d","Type":"ContainerStarted","Data":"681d7053163df1b4329cbd4df9bb835e7851ab2e75a21590a149867be606aefe"} Oct 30 19:16:47.207337 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.207314 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" event={"ID":"6fb8c089-70f4-418d-8cd4-fd2393997ccf","Type":"ContainerStarted","Data":"99dcaf0997f64d7d7f3e393c04b083b86bcd059392e4ea5690556e092265c2cd"} Oct 30 19:16:47.207422 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.207364 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" event={"ID":"6fb8c089-70f4-418d-8cd4-fd2393997ccf","Type":"ContainerStarted","Data":"2a1d5b103c91aa1c4026326d3cfdc334df60e84dbe68204ea39cdd3ebe420b0e"} Oct 30 19:16:47.207422 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.207396 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" event={"ID":"6fb8c089-70f4-418d-8cd4-fd2393997ccf","Type":"ContainerStarted","Data":"14d75d2f8598b3cb5005befda9a93a83c268b60a36aa428a0b50ebc346abe551"} Oct 30 19:16:47.209371 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.209349 2579 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee" exitCode=0 Oct 30 19:16:47.209513 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.209376 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee"} Oct 30 19:16:47.250105 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.250062 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-fjmnw" podStartSLOduration=2.412938432 podStartE2EDuration="5.250047483s" totalImagesPullingTime="2.837109051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:47.248691131 +0000 UTC m=+47.875596154" watchObservedRunningTime="2026-10-30 19:16:47.250047483 +0000 UTC m=+47.876952506" Oct 30 19:16:47.250229 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.250201 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-54p6s" podStartSLOduration=2.027744228 podStartE2EDuration="5.250193959s" totalImagesPullingTime="3.222449731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:47.230015736 +0000 UTC m=+47.856920760" watchObservedRunningTime="2026-10-30 19:16:47.250193959 +0000 UTC m=+47.877098982" Oct 30 19:16:47.272647 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.272601 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ftfsr" podStartSLOduration=2.836394712 podStartE2EDuration="4.272587143s" totalImagesPullingTime="1.353216701s" totalInitContainerRuntime="82.97573ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:47.271765842 +0000 UTC m=+47.898670867" watchObservedRunningTime="2026-10-30 19:16:47.272587143 +0000 UTC m=+47.899492168" Oct 30 19:16:47.296056 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.296014 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" podStartSLOduration=2.31948934 podStartE2EDuration="4.295998386s" totalImagesPullingTime="1.976509046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:47.294526114 +0000 UTC m=+47.921431135" watchObservedRunningTime="2026-10-30 19:16:47.295998386 +0000 UTC m=+47.922903410" Oct 30 19:16:47.997724 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:47.997645 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8988ff5d4-t8x89"] Oct 30 19:16:48.000053 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.000028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.005491 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.005471 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:16:48.005605 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.005559 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:16:48.008351 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.008330 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxtgr\"" Oct 30 19:16:48.008451 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.008381 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:16:48.008626 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.008468 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:16:48.008626 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.008537 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9vapspveagene\"" Oct 30 19:16:48.011532 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.011479 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8988ff5d4-t8x89"] Oct 30 19:16:48.108026 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.107996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwnsb\" (UniqueName: \"kubernetes.io/projected/937e5d34-bca8-4f99-bb4a-22a62c63fe73-kube-api-access-vwnsb\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.108173 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.108069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.108242 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.108182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-client-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.108242 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.108222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-client-certs\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.108331 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.108282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-tls\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.108387 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.108311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-metrics-server-audit-profiles\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.108507 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.108481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/937e5d34-bca8-4f99-bb4a-22a62c63fe73-audit-log\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.209904 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.209853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwnsb\" (UniqueName: \"kubernetes.io/projected/937e5d34-bca8-4f99-bb4a-22a62c63fe73-kube-api-access-vwnsb\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210306 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.209953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210306 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.210006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-client-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210306 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.210039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-client-certs\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210306 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.210083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-tls\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210306 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.210113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-metrics-server-audit-profiles\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210306 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.210168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/937e5d34-bca8-4f99-bb4a-22a62c63fe73-audit-log\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.210550 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.210501 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/937e5d34-bca8-4f99-bb4a-22a62c63fe73-audit-log\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.211387 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.211360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.213879 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.212644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-metrics-server-audit-profiles\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.214047 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.214011 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-client-certs\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.214476 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.214452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-tls\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.214476 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.214463 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-client-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.217602 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.217580 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bpk2j" event={"ID":"471d5763-db1f-46f7-99fe-d55f45fa9a76","Type":"ContainerStarted","Data":"be3c6ad8c1f9b9bed27b4b80fa85bc0f6c51f49311435962c94a0629ca8e0343"} Oct 30 19:16:48.220556 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.220533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwnsb\" (UniqueName: \"kubernetes.io/projected/937e5d34-bca8-4f99-bb4a-22a62c63fe73-kube-api-access-vwnsb\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.233335 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.233289 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bpk2j" podStartSLOduration=2.29824797 podStartE2EDuration="6.233275305s" totalImagesPullingTime="3.935027335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:48.233094146 +0000 UTC m=+48.859999168" watchObservedRunningTime="2026-10-30 19:16:48.233275305 +0000 UTC m=+48.860180329" Oct 30 19:16:48.315104 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.314775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:16:48.339116 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.339089 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-slbxc"] Oct 30 19:16:48.342002 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.341981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.354477 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.352637 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w7ctm\"" Oct 30 19:16:48.354477 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.353496 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:16:48.354477 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.354410 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-slbxc"] Oct 30 19:16:48.377498 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.377441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:48.411542 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.411509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfa4a876-df8d-45b5-aa59-37bf104db694-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.411662 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.411640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5q52\" (UniqueName: \"kubernetes.io/projected/bfa4a876-df8d-45b5-aa59-37bf104db694-kube-api-access-l5q52\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.482904 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.482859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:48.513061 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.513034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfa4a876-df8d-45b5-aa59-37bf104db694-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.513197 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.513106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q52\" (UniqueName: \"kubernetes.io/projected/bfa4a876-df8d-45b5-aa59-37bf104db694-kube-api-access-l5q52\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.516155 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.516127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfa4a876-df8d-45b5-aa59-37bf104db694-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.522467 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.522446 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q52\" (UniqueName: \"kubernetes.io/projected/bfa4a876-df8d-45b5-aa59-37bf104db694-kube-api-access-l5q52\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.583940 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.583875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:48.599930 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.599905 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8988ff5d4-t8x89"] Oct 30 19:16:48.604308 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:48.604279 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937e5d34_bca8_4f99_bb4a_22a62c63fe73.slice/crio-c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a WatchSource:0}: Error finding container c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a: Status 404 returned error can't find the container with id c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a Oct 30 19:16:48.658754 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.658728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:48.787532 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.787509 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x"] Oct 30 19:16:48.790885 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.790855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.794676 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.794658 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 19:16:48.794744 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.794733 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-78j2g\"" Oct 30 19:16:48.794819 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.794807 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 19:16:48.794970 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.794957 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 19:16:48.795010 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.794950 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 19:16:48.808931 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.808913 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x"] Oct 30 19:16:48.817654 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.817617 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 19:16:48.820982 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.820964 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 19:16:48.918409 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ph5t\" (UniqueName: \"kubernetes.io/projected/f438e609-1285-4d67-8622-7b5022bdee31-kube-api-access-8ph5t\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918482 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-metrics-client-ca\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918482 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-federate-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918482 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918584 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918584 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-serving-certs-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918584 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.918584 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.918583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:48.975226 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:48.975201 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-slbxc"] Oct 30 19:16:48.977767 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:48.977745 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa4a876_df8d_45b5_aa59_37bf104db694.slice/crio-2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d WatchSource:0}: Error finding container 2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d: Status 404 returned error can't find the container with id 2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d Oct 30 19:16:49.019200 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-serving-certs-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019313 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019389 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019443 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ph5t\" (UniqueName: \"kubernetes.io/projected/f438e609-1285-4d67-8622-7b5022bdee31-kube-api-access-8ph5t\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019443 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-metrics-client-ca\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019543 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-federate-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019543 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.019543 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.019520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.020031 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.020002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-serving-certs-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.020275 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.020250 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-metrics-client-ca\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.020357 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.020309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.022093 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.022069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.022396 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.022369 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-federate-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.022617 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.022596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.022617 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.022607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.027552 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.027528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ph5t\" (UniqueName: \"kubernetes.io/projected/f438e609-1285-4d67-8622-7b5022bdee31-kube-api-access-8ph5t\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.077784 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.077762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.117984 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.117962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:16:49.182642 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.182613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.223688 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.223648 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c4467959-2ccth" event={"ID":"45afd0c1-8540-4a90-a7ee-1db0805c9e2c","Type":"ContainerStarted","Data":"d39744170965de47e2c3f6a1556277dccbc9ba03e34a7524db4615400393a169"} Oct 30 19:16:49.227808 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.227781 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2"} Oct 30 19:16:49.227939 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.227849 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e"} Oct 30 19:16:49.227939 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.227908 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784"} Oct 30 19:16:49.227939 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.227934 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e"} Oct 30 19:16:49.228091 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.227961 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98"} Oct 30 19:16:49.229620 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.229596 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" event={"ID":"937e5d34-bca8-4f99-bb4a-22a62c63fe73","Type":"ContainerStarted","Data":"c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a"} Oct 30 19:16:49.231257 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.231234 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" event={"ID":"bfa4a876-df8d-45b5-aa59-37bf104db694","Type":"ContainerStarted","Data":"2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d"} Oct 30 19:16:49.242023 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.241980 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56c4467959-2ccth" podStartSLOduration=1.0778743 podStartE2EDuration="4.241964732s" totalImagesPullingTime="3.164090432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:49.241637228 +0000 UTC m=+49.868542252" watchObservedRunningTime="2026-10-30 19:16:49.241964732 +0000 UTC m=+49.868869754" Oct 30 19:16:49.405168 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.405094 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x"] Oct 30 19:16:49.408891 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.408115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.408891 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:49.408456 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf438e609_1285_4d67_8622_7b5022bdee31.slice/crio-a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa WatchSource:0}: Error finding container a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa: Status 404 returned error can't find the container with id a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa Oct 30 19:16:49.519738 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.519681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.843563 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:49.843464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:50.240628 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:50.240594 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerStarted","Data":"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df"} Oct 30 19:16:50.244063 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:50.244033 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" event={"ID":"937e5d34-bca8-4f99-bb4a-22a62c63fe73","Type":"ContainerStarted","Data":"b05b38cd12f7303b1e6354a411818ddf048d3f9b63f6fcd7a6331b819dd5ec3a"} Oct 30 19:16:50.246176 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:50.246116 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" event={"ID":"f438e609-1285-4d67-8622-7b5022bdee31","Type":"ContainerStarted","Data":"a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa"} Oct 30 19:16:50.288827 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:50.288780 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.542058178 podStartE2EDuration="6.288768943s" totalImagesPullingTime="3.735509002s" totalInitContainerRuntime="11.201763ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:50.285711019 +0000 UTC m=+50.912616042" watchObservedRunningTime="2026-10-30 19:16:50.288768943 +0000 UTC m=+50.915673963" Oct 30 19:16:50.320105 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:50.318767 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" podStartSLOduration=1.866734035 podStartE2EDuration="3.318750247s" totalImagesPullingTime="1.452016212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:50.316964176 +0000 UTC m=+50.943869199" watchObservedRunningTime="2026-10-30 19:16:50.318750247 +0000 UTC m=+50.945655272" Oct 30 19:16:51.250761 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.250719 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" event={"ID":"bfa4a876-df8d-45b5-aa59-37bf104db694","Type":"ContainerStarted","Data":"7ea2647a417122b9ca05dd2747ffa651c2e8e163bee80e08831c744ab9b6442b"} Oct 30 19:16:51.251639 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.251617 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:51.254196 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.254166 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" event={"ID":"f438e609-1285-4d67-8622-7b5022bdee31","Type":"ContainerStarted","Data":"6e89a2d3675ddffddc5766fcd223290f32304596605ae3383d816b3153da5dec"} Oct 30 19:16:51.254284 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.254235 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" event={"ID":"f438e609-1285-4d67-8622-7b5022bdee31","Type":"ContainerStarted","Data":"955bf890872b6163c531fa1782e37f507490f9bc2a816e5bc5e6fedde77ff744"} Oct 30 19:16:51.254326 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.254264 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" event={"ID":"f438e609-1285-4d67-8622-7b5022bdee31","Type":"ContainerStarted","Data":"bcd1570339b4d7b5ab259dcd127e039edeae021abbc601cf86d1f0721fecf183"} Oct 30 19:16:51.257283 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.257265 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:16:51.266967 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.266920 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" podStartSLOduration=1.834787849 podStartE2EDuration="3.266907873s" totalImagesPullingTime="1.432120024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:51.266224375 +0000 UTC m=+51.893129396" watchObservedRunningTime="2026-10-30 19:16:51.266907873 +0000 UTC m=+51.893812894" Oct 30 19:16:51.302015 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:51.301971 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" podStartSLOduration=1.6571087169999998 podStartE2EDuration="3.301955407s" totalImagesPullingTime="1.64484669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:51.300699874 +0000 UTC m=+51.927604896" watchObservedRunningTime="2026-10-30 19:16:51.301955407 +0000 UTC m=+51.928860430" Oct 30 19:16:53.040978 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:53.040950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:53.152996 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:53.152973 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:54.111840 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:54.111816 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:16:55.406384 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:55.406353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:55.509079 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:55.509056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:55.552567 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:55.552536 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:55.552677 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:55.552610 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:55.556936 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:55.556919 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:56.274785 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:56.274758 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:16:57.168977 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.168938 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5d44cd94-pfbqm"] Oct 30 19:16:57.174199 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.174173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.181005 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.180282 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5d44cd94-pfbqm"] Oct 30 19:16:57.199842 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.199819 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 19:16:57.220833 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.220809 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fjmnw" Oct 30 19:16:57.294612 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.294583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.294757 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.294621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.294757 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.294722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.294888 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.294801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.294888 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.294836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.295034 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.294992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.295194 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.295169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.295261 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.295225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.395948 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.395909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.395948 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.395945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396156 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396251 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396327 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396431 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396487 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396487 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396476 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396578 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.396755 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.396729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.397228 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.397201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.397361 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.397341 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.397627 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.397607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.398984 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.398963 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.402492 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.402473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.407014 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.406991 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.487549 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.487515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:16:57.762885 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:57.762729 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5d44cd94-pfbqm"] Oct 30 19:16:57.779731 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:16:57.779703 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d0014b_03d6_4271_8b1b_3749435c511f.slice/crio-4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1 WatchSource:0}: Error finding container 4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1: Status 404 returned error can't find the container with id 4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1 Oct 30 19:16:58.213921 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:58.213815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.280983 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:58.280945 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d44cd94-pfbqm" event={"ID":"b3d0014b-03d6-4271-8b1b-3749435c511f","Type":"ContainerStarted","Data":"a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4"} Oct 30 19:16:58.281109 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:58.281005 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d44cd94-pfbqm" event={"ID":"b3d0014b-03d6-4271-8b1b-3749435c511f","Type":"ContainerStarted","Data":"4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1"} Oct 30 19:16:58.307829 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:58.304367 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d5d44cd94-pfbqm" podStartSLOduration=1.304349441 podStartE2EDuration="1.304349441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:58.302282807 +0000 UTC m=+58.929187832" watchObservedRunningTime="2026-10-30 19:16:58.304349441 +0000 UTC m=+58.931254463" Oct 30 19:16:58.325009 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:58.324982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:59.087488 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:59.087457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:59.198954 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:16:59.198929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:04.196693 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.196664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:04.317329 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.317303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:04.661890 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.661823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:04.661890 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.661884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:04.665034 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.665020 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:17:04.666020 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.666005 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:17:04.674148 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.674130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:04.674148 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.674146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:04.762492 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.762466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:04.765360 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.765345 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:17:04.775898 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.775881 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:17:04.785965 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.785946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:04.906987 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.906968 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\"" Oct 30 19:17:04.915090 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.915033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:04.948027 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.948002 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\"" Oct 30 19:17:04.950172 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.950152 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\"" Oct 30 19:17:04.956287 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.956268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:04.958533 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:04.958511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:05.182674 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:05.182650 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5sqtd"] Oct 30 19:17:05.185046 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:17:05.185025 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b654842_b7a7_4e8d_8c92_8729111d0373.slice/crio-8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6 WatchSource:0}: Error finding container 8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6: Status 404 returned error can't find the container with id 8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6 Oct 30 19:17:05.233394 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:05.233373 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tfj5g"] Oct 30 19:17:05.237013 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:05.236983 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lc88s"] Oct 30 19:17:05.237907 ip-10-0-138-233 kubenswrapper[2579]: W1030 19:17:05.237882 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06348567_413f_417b_a78b_a10d7ff83ff4.slice/crio-3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21 WatchSource:0}: Error finding container 3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21: Status 404 returned error can't find the container with id 3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21 Oct 30 19:17:05.303528 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:05.303504 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc88s" event={"ID":"06348567-413f-417b-a78b-a10d7ff83ff4","Type":"ContainerStarted","Data":"3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21"} Oct 30 19:17:05.304628 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:05.304603 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tfj5g" event={"ID":"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f","Type":"ContainerStarted","Data":"e583ac9f2513029b7a03559e9950ed68703dec1df40a56e8d638c047734587c3"} Oct 30 19:17:05.305707 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:05.305688 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5sqtd" event={"ID":"6b654842-b7a7-4e8d-8c92-8729111d0373","Type":"ContainerStarted","Data":"8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6"} Oct 30 19:17:07.318726 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.318684 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc88s" event={"ID":"06348567-413f-417b-a78b-a10d7ff83ff4","Type":"ContainerStarted","Data":"57808357e554804adaf2ea4f13862901e4a61c6c07ae63ec5659f313a702b0de"} Oct 30 19:17:07.319293 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.318762 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lc88s" event={"ID":"06348567-413f-417b-a78b-a10d7ff83ff4","Type":"ContainerStarted","Data":"592fcf439f8b053961dc8826f199e654a7088b81c0c4f2137873733722bb1093"} Oct 30 19:17:07.336255 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.336189 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lc88s" podStartSLOduration=66.224241293 podStartE2EDuration="1m7.336172218s" totalImagesPullingTime="1.111930925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:07.334287893 +0000 UTC m=+67.961192917" watchObservedRunningTime="2026-10-30 19:17:07.336172218 +0000 UTC m=+67.963077241" Oct 30 19:17:07.466083 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.466039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:07.488204 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.488174 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:07.488922 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.488896 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:07.495307 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.495286 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:07.572110 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.572019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:07.675213 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:07.675181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:08.315769 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:08.315735 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:08.316532 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:08.316508 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:08.328090 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:08.328066 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:08.383530 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:08.383504 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56c4467959-2ccth"] Oct 30 19:17:09.329826 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:09.329783 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tfj5g" event={"ID":"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f","Type":"ContainerStarted","Data":"cd9d22877fb9ecbaabf1308b9e69334aa1a859acd3111f52a4aaec3ecbbf3006"} Oct 30 19:17:09.330292 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:09.330177 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:09.346463 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:09.346412 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tfj5g" podStartSLOduration=67.132854384 podStartE2EDuration="1m10.346395722s" totalImagesPullingTime="3.213541338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:09.345775397 +0000 UTC m=+69.972680420" watchObservedRunningTime="2026-10-30 19:17:09.346395722 +0000 UTC m=+69.973300744" Oct 30 19:17:09.419427 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:09.419395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:09.552542 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:09.552504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:09.994535 ip-10-0-138-233 kubenswrapper[2579]: I1030 19:17:09.994478 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:17:09.994684 ip-10-0-138-233 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:17:10.003428 ip-10-0-138-233 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:17:10.003593 ip-10-0-138-233 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:17:10.003762 ip-10-0-138-233 systemd[1]: kubelet.service: Consumed 4.069s CPU time. Oct 30 19:17:10.011698 ip-10-0-138-233 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:17:10.072759 ip-10-0-138-233 kubenswrapper[6514]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:17:10.072759 ip-10-0-138-233 kubenswrapper[6514]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:17:10.072759 ip-10-0-138-233 kubenswrapper[6514]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:17:10.072759 ip-10-0-138-233 kubenswrapper[6514]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:17:10.074712 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074642 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.074712 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074712 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074715 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074719 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074721 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074724 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074727 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074729 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074732 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074735 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074737 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074740 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074742 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074745 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074749 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074752 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074755 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074758 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074760 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074763 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.074782 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074765 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074768 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074770 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074773 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074775 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074784 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074787 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074790 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074793 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074796 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074798 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074801 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074803 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074806 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074808 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074810 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074813 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074815 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074818 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.075272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074820 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074822 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074825 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074827 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074830 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074832 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074834 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074837 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074840 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074843 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074845 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074848 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074850 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074856 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074858 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074874 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074878 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074882 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074885 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074889 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.075733 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074892 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074894 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074897 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074901 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074906 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074909 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074912 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074914 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074917 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074919 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074922 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074924 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074927 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074929 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074932 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074935 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074937 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074940 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074943 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.076241 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074946 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074949 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074951 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074954 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074956 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074959 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074964 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074967 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074969 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074972 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074974 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074977 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074979 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074982 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074985 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074988 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074990 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074993 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074995 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.074998 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.076706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075001 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075003 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075013 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075022 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075025 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075027 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075030 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075032 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075770 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075777 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075781 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075784 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075787 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075792 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075796 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075799 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075802 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075805 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.077209 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075808 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075811 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075813 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075816 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075819 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075821 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075824 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075826 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075830 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075833 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075835 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075838 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075840 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075843 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075846 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075849 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075851 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075854 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075856 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075873 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.077650 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075876 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075879 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075881 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075884 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075887 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075889 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075892 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075895 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075898 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075900 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075903 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075906 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075909 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075911 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075914 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075916 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075919 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075922 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075924 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075927 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.078147 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075929 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075936 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075939 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075942 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075944 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075947 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075950 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075953 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075955 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075958 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075961 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075963 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075967 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075970 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075972 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075975 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075977 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075980 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075983 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075986 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.078659 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075989 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075992 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075994 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.075997 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076000 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076002 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076005 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076008 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076011 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076013 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076016 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076018 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076021 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076023 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076026 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076029 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076031 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076034 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076037 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.079182 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076039 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076042 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076045 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076047 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076050 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076053 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076055 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076058 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076060 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076063 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076065 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076068 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076071 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076073 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076076 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076079 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076084 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076575 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076581 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076584 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.079640 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076588 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076590 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076593 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076595 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076598 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076600 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076603 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076606 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076609 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076611 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076614 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076617 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076620 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076622 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076625 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076627 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076630 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076632 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076635 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.080160 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076637 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076640 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076643 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076645 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076647 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076650 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076654 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076658 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076662 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076667 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076669 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076672 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076675 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076677 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076680 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076682 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076685 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076688 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076691 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.080641 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076693 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076696 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076699 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076702 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076705 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076707 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076710 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076712 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076715 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076718 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076721 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076724 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076726 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076729 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076731 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076733 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076736 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076738 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076742 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.081126 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076745 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076747 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076750 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076755 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076758 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076760 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076763 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076765 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076768 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076770 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076773 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076775 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076778 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076782 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076785 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076788 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076791 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076793 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076796 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.081573 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076798 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076800 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076803 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076805 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076808 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076810 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076813 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076815 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076818 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076820 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076822 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076825 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076829 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076832 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076834 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076837 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076840 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076843 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076846 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076848 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.082044 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076851 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076853 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076855 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076858 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076872 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076875 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.076878 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076968 6514 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076975 6514 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076981 6514 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076987 6514 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076991 6514 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076994 6514 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.076998 6514 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077002 6514 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077005 6514 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077008 6514 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077011 6514 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077015 6514 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077018 6514 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077020 6514 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077023 6514 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:17:10.082515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077026 6514 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077029 6514 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077034 6514 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077038 6514 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077040 6514 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077043 6514 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077047 6514 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077051 6514 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077055 6514 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077057 6514 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077060 6514 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077063 6514 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077066 6514 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077069 6514 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077072 6514 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077075 6514 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077078 6514 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077082 6514 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077085 6514 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077088 6514 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077091 6514 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077095 6514 flags.go:64] FLAG: --enable-server="true" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077098 6514 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077102 6514 flags.go:64] FLAG: --event-burst="100" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077105 6514 flags.go:64] FLAG: --event-qps="50" Oct 30 19:17:10.083063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077108 6514 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077111 6514 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077114 6514 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077118 6514 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077121 6514 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077124 6514 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077127 6514 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077130 6514 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077133 6514 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077137 6514 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077140 6514 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077143 6514 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077145 6514 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077150 6514 flags.go:64] FLAG: --feature-gates="" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077154 6514 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077157 6514 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077160 6514 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077163 6514 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077166 6514 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077169 6514 flags.go:64] FLAG: --help="false" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077172 6514 flags.go:64] FLAG: --hostname-override="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077175 6514 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077178 6514 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:17:10.084032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077181 6514 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077184 6514 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077188 6514 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077191 6514 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077194 6514 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077197 6514 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077200 6514 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077203 6514 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077206 6514 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077209 6514 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077212 6514 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077215 6514 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077220 6514 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077223 6514 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077226 6514 flags.go:64] FLAG: --lock-file="" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077229 6514 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077232 6514 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077235 6514 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077241 6514 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077243 6514 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077246 6514 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077249 6514 flags.go:64] FLAG: --logging-format="text" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077253 6514 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077275 6514 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:17:10.084970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077279 6514 flags.go:64] FLAG: --manifest-url="" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077282 6514 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077287 6514 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077290 6514 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077295 6514 flags.go:64] FLAG: --max-pods="110" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077298 6514 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077301 6514 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077304 6514 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077307 6514 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077310 6514 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077314 6514 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077317 6514 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077323 6514 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077326 6514 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077329 6514 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077336 6514 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077340 6514 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077343 6514 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077346 6514 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077349 6514 flags.go:64] FLAG: --port="10250" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077352 6514 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077354 6514 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0715b9e8fe625cdcd" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077358 6514 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077361 6514 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077364 6514 flags.go:64] FLAG: --register-node="true" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077367 6514 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:17:10.085540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077371 6514 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077374 6514 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077377 6514 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077380 6514 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077385 6514 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077388 6514 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077391 6514 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077394 6514 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077397 6514 flags.go:64] FLAG: --runonce="false" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077400 6514 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077403 6514 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077406 6514 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077408 6514 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077411 6514 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077414 6514 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077417 6514 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077420 6514 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077423 6514 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077426 6514 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077429 6514 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077432 6514 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077436 6514 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077439 6514 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077442 6514 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077447 6514 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:17:10.086176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077450 6514 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077453 6514 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077456 6514 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077459 6514 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077462 6514 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077465 6514 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077468 6514 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077471 6514 flags.go:64] FLAG: --v="2" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077475 6514 flags.go:64] FLAG: --version="false" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077479 6514 flags.go:64] FLAG: --vmodule="" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077483 6514 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077488 6514 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077617 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077621 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077624 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077626 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077629 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077632 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077634 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077636 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077639 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077642 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.086755 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077644 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077647 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077649 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077652 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077655 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077658 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077661 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077663 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077667 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077669 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077672 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077675 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077677 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077679 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077682 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077684 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077687 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077690 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077692 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077695 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.087361 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077697 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077702 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077706 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077709 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077712 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077715 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077718 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077720 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077723 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077726 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077728 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077731 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077734 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077736 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077739 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077741 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077743 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077746 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077748 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.087914 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077751 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077754 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077757 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077761 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077763 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077765 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077768 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077770 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077773 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077776 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077778 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077781 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077783 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077785 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077790 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077792 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077795 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077797 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077800 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.088426 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077803 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077805 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077809 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077812 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077815 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077817 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077820 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077822 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077825 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077827 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077830 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077833 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077835 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077838 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077840 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077843 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077847 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077850 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077852 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.089047 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077855 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077857 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077887 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077890 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077893 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077895 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077898 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077900 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077904 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077907 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077909 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077912 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077915 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077917 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077920 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077923 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077926 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077928 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.089498 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077931 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.077958 6514 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.077996 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078000 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078003 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078006 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078009 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078012 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078014 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078017 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078019 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078023 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078031 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078033 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078038 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078042 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.090024 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078045 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078048 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078051 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078054 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078057 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078059 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078062 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078066 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078069 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078071 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078074 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078076 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078079 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078082 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078085 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078087 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078090 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078092 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078095 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078097 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.090459 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078100 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078102 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078105 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078108 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078110 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078113 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078115 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078119 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078121 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078124 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078127 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078130 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078132 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078135 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078138 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078140 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078144 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078147 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078151 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.091092 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078155 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078158 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078160 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078163 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078165 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078168 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078170 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078173 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078175 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078178 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078180 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078183 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078186 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078188 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078190 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078193 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078195 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078198 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078201 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078203 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.091627 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078208 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078210 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078212 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078215 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078218 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078221 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078223 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078226 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078228 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078231 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078233 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078236 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078240 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078243 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078245 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078248 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078250 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078253 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078255 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.092132 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078258 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078260 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078263 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078266 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078268 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078270 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078273 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078275 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078278 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078281 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078283 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078286 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078288 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.078293 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.081661 6514 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.081676 6514 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.081693 6514 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.081698 6514 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081751 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081756 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.092586 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081759 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081762 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081765 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081769 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081771 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081774 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081777 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081780 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081783 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081785 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081788 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081791 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081794 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081797 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081800 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081802 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081805 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081807 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081810 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081813 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.093076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081815 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081818 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081821 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081824 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081827 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081829 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081832 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081835 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081837 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081841 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081843 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081846 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081848 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081851 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081853 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081856 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081858 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081880 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081884 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081889 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.093534 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081893 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081896 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081899 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081902 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081904 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081908 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081911 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081913 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081916 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081918 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081922 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081924 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081927 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081929 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081932 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081935 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081937 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081940 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081943 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.094009 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081946 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081949 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081952 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081954 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081957 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081960 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081962 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081965 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081968 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081970 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081973 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081975 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081978 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081980 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081983 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081985 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081988 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081990 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081993 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.094442 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081995 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.081998 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082000 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082005 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082009 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082012 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082015 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082018 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082021 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082023 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082026 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082029 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082033 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082036 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082039 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082042 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082044 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082047 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082049 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.094894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082052 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082055 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082058 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082060 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082062 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082065 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082067 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.082094 6514 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082224 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082228 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082231 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082234 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082237 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082239 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082242 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082245 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.095461 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082248 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082251 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082253 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082256 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082259 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082262 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082264 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082267 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082269 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082272 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082275 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082277 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082280 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082282 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082285 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082287 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082290 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082293 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082295 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.095930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082298 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082300 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082303 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082305 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082309 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082313 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082316 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082319 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082321 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082323 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082326 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082329 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082332 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082334 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082337 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082339 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082342 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082344 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082347 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.096408 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082349 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082352 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082354 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082357 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082359 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082361 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082364 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082366 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082369 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082371 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082374 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082376 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082379 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082381 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082383 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082386 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082388 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082390 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082393 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.096854 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082396 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082398 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082400 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082403 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082405 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082408 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082410 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082413 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082415 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082418 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082420 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082423 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082425 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082428 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082431 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082433 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082436 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082438 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082442 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082445 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.097352 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082448 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082451 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082453 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082456 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082459 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082461 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082464 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082467 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082469 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082472 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082474 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082477 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082479 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082482 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082484 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082487 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082489 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082492 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082494 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082496 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.097820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082499 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.082522 6514 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082550 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082554 6514 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082558 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082561 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082564 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082567 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082569 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082572 6514 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082575 6514 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082578 6514 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082582 6514 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082585 6514 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082588 6514 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082591 6514 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:10.098306 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082593 6514 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082596 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082599 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082601 6514 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082604 6514 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082606 6514 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082609 6514 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082612 6514 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082614 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082617 6514 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082619 6514 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082622 6514 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082624 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082627 6514 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082629 6514 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082632 6514 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082634 6514 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082638 6514 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082640 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082646 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:10.098721 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082648 6514 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082651 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082653 6514 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082656 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082658 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082660 6514 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082664 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082668 6514 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082670 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082673 6514 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082675 6514 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082678 6514 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082681 6514 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082683 6514 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082686 6514 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082688 6514 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082691 6514 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082693 6514 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082696 6514 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:10.099487 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082699 6514 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082701 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082704 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082706 6514 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082709 6514 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082711 6514 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082714 6514 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082716 6514 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082719 6514 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082721 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082724 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082728 6514 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082733 6514 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082737 6514 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082739 6514 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082742 6514 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082744 6514 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082747 6514 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082749 6514 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:10.100084 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082753 6514 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082756 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082759 6514 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082761 6514 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082764 6514 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082766 6514 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082769 6514 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082771 6514 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082774 6514 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082777 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082779 6514 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082782 6514 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082784 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082787 6514 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082789 6514 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082792 6514 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082794 6514 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082797 6514 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082799 6514 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:10.100558 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082802 6514 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082804 6514 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082807 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082809 6514 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082811 6514 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082814 6514 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082818 6514 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082821 6514 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082823 6514 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082826 6514 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082828 6514 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082831 6514 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082833 6514 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082836 6514 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:10.082839 6514 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.082900 6514 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.085332 6514 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.085381 6514 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.085560 6514 server.go:1014] "Starting client certificate rotation" Oct 30 19:17:10.101031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.085694 6514 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.085729 6514 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:59 +0000 UTC" deadline="2028-08-07 00:06:15.82822607 +0000 UTC" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.085755 6514 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15508h49m5.742472894s" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.086002 6514 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.087013 6514 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.088127 6514 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.089937 6514 log.go:25] "Validated CRI v1 image API" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.090631 6514 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:17:10.101714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.092790 6514 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b04e1b63-d6a8-4d39-a43f-bd79372f3797:/dev/nvme0n1p3 d604349d-da9e-42eb-bab5-7290335ac519:/dev/nvme0n1p4] Oct 30 19:17:10.102100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.092807 6514 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c4a8b38ed11b915f3d6f1316cabb3f69aa1adb696d57c5d8e1438c7c7f6c598/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c4a8b38ed11b915f3d6f1316cabb3f69aa1adb696d57c5d8e1438c7c7f6c598/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21/userdata/shm major:0 minor:441 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1/userdata/shm major:0 minor:425 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6/userdata/shm major:0 minor:438 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e583ac9f2513029b7a03559e9950ed68703dec1df40a56e8d638c047734587c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e583ac9f2513029b7a03559e9950ed68703dec1df40a56e8d638c047734587c3/userdata/shm major:0 minor:440 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06348567-413f-417b-a78b-a10d7ff83ff4/volumes/kubernetes.io~projected/kube-api-access-9sgj7:{mountpoint:/var/lib/kubelet/pods/06348567-413f-417b-a78b-a10d7ff83ff4/volumes/kubernetes.io~projected/kube-api-access-9sgj7 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06348567-413f-417b-a78b-a10d7ff83ff4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/06348567-413f-417b-a78b-a10d7ff83ff4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15140033-be3b-4729-9a78-3a38b3d1ef48/volumes/kubernetes.io~projected/kube-api-access-v4dn2:{mountpoint:/var/lib/kubelet/pods/15140033-be3b-4729-9a78-3a38b3d1ef48/volumes/kubernetes.io~projected/kube-api-access-v4dn2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab/volumes/kubernetes.io~projected/kube-api-access-6qf2c:{mountpoint:/var/lib/kubelet/pods/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab/volumes/kubernetes.io~projected/kube-api-access-6qf2c major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0/volumes/kubernetes.io~projected/kube-api-access-tw2pn:{mountpoint:/var/lib/kubelet/pods/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0/volumes/kubernetes.io~projected/kube-api-access-tw2pn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26bc4ca2-8079-4560-8891-dc52f796c72f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/26bc4ca2-8079-4560-8891-dc52f796c72f/volumes/kubernetes.io~secret/agent-certs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~projected/kube-api-access-sc9t4:{mountpoint:/var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~projected/kube-api-access-sc9t4 major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c9c1b81-6bc0-406b-a0c1-a370afa909b0/volumes/kubernetes.io~projected/kube-api-access-m2d7j:{mountpoint:/var/lib/kubelet/pods/3c9c1b81-6bc0-406b-a0c1-a370afa909b0/volumes/kubernetes.io~projected/kube-api-access-m2d7j major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c9c1b81-6bc0-406b-a0c1-a370afa909b0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3c9c1b81-6bc0-406b-a0c1-a370afa909b0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~projected/kube-api-access-hn2td:{mountpoint:/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~projected/kube-api-access-hn2td major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471d5763-db1f-46f7-99fe-d55f45fa9a76/volumes/kubernetes.io~projected/kube-api-access-tlcq4:{mountpoint:/var/lib/kubelet/pods/471d5763-db1f-46f7-99fe-d55f45fa9a76/volumes/kubernetes.io~projected/kube-api-access-tlcq4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471d5763-db1f-46f7-99fe-d55f45fa9a76/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/471d5763-db1f-46f7-99fe-d55f45fa9a76/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53f48b9b-ec52-44f6-a537-e6e585762c7e/volumes/kubernetes.io~projected/kube-api-access-46jgh:{mountpoint:/var/lib/kubelet/pods/53f48b9b-ec52-44f6-a537-e6e585762c7e/volumes/kubernetes.io~projected/kube-api-access-46jgh major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b654842-b7a7-4e8d-8c92-8729111d0373/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6b654842-b7a7-4e8d-8c92-8729111d0373/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~projected/kube-api-access-4g88x:{mountpoint:/var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~projected/kube-api-access-4g88x major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volumes/kubernetes.io~projected/kube-api-access-drbpg:{mountpoint:/var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volumes/kubernetes.io~projected/kube-api-access-drbpg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/862b83b6-5666-4f55-a276-4cb674c0b4a8/volumes/kubernetes.io~projected/kube-api-access-cthhb:{mountpoint:/var/lib/kubelet/pods/862b83b6-5666-4f55-a276-4cb674c0b4a8/volumes/kubernetes.io~projected/kube-api-access-cthhb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~projected/kube-api-access-vwnsb:{mountpoint:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~projected/kube-api-access-vwnsb major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ef62b57-79b3-4002-ba56-d41d3b74ded9/volumes/kubernetes.io~projected/kube-api-access-2nhpt:{mountpoint:/var/lib/kubelet/pods/9ef62b57-79b3-4002-ba56-d41d3b74ded9/volumes/kubernetes.io~projected/kube-api-access-2nhpt major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~projected/kube-api-access-7c4cf:{mountpoint:/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~projected/kube-api-access-7c4cf major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa4a876-df8d-45b5-aa59-37bf104db694/volumes/kubernetes.io~projected/kube-api-access-l5q52:{mountpoint:/var/lib/kubelet/pods/bfa4a876-df8d-45b5-aa59-37bf104db694/volumes/kubernetes.io~projected/kube-api-access-l5q52 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa4a876-df8d-45b5-aa59-37bf104db694/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bfa4a876-df8d-45b5-aa59-37bf104db694/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c346bfd4-9367-4a93-a44e-022b444c9f8c/volumes/kubernetes.io~projected/kube-api-access-chvlc:{mountpoint:/var/lib/kubelet/pods/c346bfd4-9367-4a93-a44e-022b444c9f8c/volumes/kubernetes.io~projected/kube-api-access-chvlc major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c346bfd4-9367-4a93-a44e-022b444c9f8c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c346bfd4-9367-4a93-a44e-022b444c9f8c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/alertmanager/9 major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/config-reloader/6 major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~empty-dir/config-out major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~projected/kube-api-access-sc85n:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~projected/kube-api-access-sc85n major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~projected/tls-assets major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/config-volume major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/web-config major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~projected/kube-api-access-stb86:{mountpoint:/var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~projected/kube-api-access-stb86 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f/volumes/kubernetes.io~projected/kube-api-access-th98d:{mountpoint:/var/lib/kubelet/pods/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f/volumes/kubernetes.io~projected/kube-api-access-th98d major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~projected/kube-api-access-8ph5t:{mountpoint:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~projected/kube-api-access-8ph5t major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~projected/kube-api-access-mclx4:{mountpoint:/var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~projected/kube-api-access-mclx4 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:227 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bd859417504ac93fe5865e0810fbcf727145a1735a27c49196d6352939da197b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/616cabc012d8765e8e7ac4f74fced73e3a17b020da862fdbef1b92ac0ed2a01c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3c01a64f9b47b12465115cf604f0b3bee4a2af6a96191397a96a7b5df1e1a136/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d7379330599a537e0981d09330410ea8fd7f76e7aefe7bc8d322f9895ff3d2ef/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0b0ccff0801f1698167400067913e8a95ac580bfa100d2b4a73e8941589106e4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/608418a79632bdeeb3c27b3897be5d3d83b3ee528a7352767ae316370ecb6c9c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/40a3c1e4896a3b91fdc68c21234c50bf2c30a517d2ff9b81c840846966798faa/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f157c09b25d2054dc2592e897512bad618d7a5e0f331fc7d58a6ce4e9dc0d262/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/29559f41412364d288603458e57799100957ac1bbf7793db191137580aa688a0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9c939e07af55b02fe1a88cc5543460628abf71b494022d32248ab926126ae55a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/31aa8d9fb28ba7e8af64d910588ee8eb1ec74cb94901d726a67359f898c76a26/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ddae9b37d69f16b09c72810f10c0a8094ae8b8102e827441cca76f6084fb18ea/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2959df4e53e5037ad56f5263636d97e0cc2d15612f3c4ea76ce8ce72af8eab38/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/08fd9c305744eae4ff0234c1cd0137b85b3454af4df108ab6d4718dd91d8a6f2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/436a12a018ca1a872f5f9619141017b8cb5710d644428bd61300aeff5bf48dc9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7e11ac217057cb3a0d55ea859e75d4d05b3ab618c0e715b537f4ea646a547921/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/60bfe85c49b344325a74b16ca67f3242910931a30080dd787fed174f7baf8b8f/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/9e20a9d5d506361ce6ec0f8844d72ef6af5a0ed1c294e958f9167c4e20952951/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ff88bf851d3e967d5a5a0b229013d41e51a149f61ac33c40d6bd82fcded20de3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fae0269862e101404b4d3d7cbf8e6fbcdbdd9db53689e33118e48e7b79827c18/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a8d67277e7086169b602922784e89e94b0587ce00a7ba1db0afc9b9915ee9724/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d175b6bd9dd27eeb9a0d4c7ee81bc9a066a2a6f8e59e45bbe89db8e988330228/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f40f08f7b8a4902d8710038cb9cb9dd011323ecb2d184932f6ef481f98dd110b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d312570ce97432f17fabd154e4d54914801634d0358a89ff55ddcac8749bbbcc/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c2bc12d5aa8565c660dbd18e87b35ffb8e5c40e9d3ac35b30bb4dbdb95675b7c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/7dfd82e0dc8316ac3063c01041b9e2d4768429202b10ea857f5b5ec4f9dd2433/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/cbea7f2f8ea452bd6faa064891ed27c0dbed9c938f483864117097ebc39c8d73/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/f4c9d93081566c527a0ed8cecdd514f3bd5e77670cd34717a68b4a510b5b4d58/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/3d1784d6eb3770a87d95c27cf21c117e163e8e579fe2de02cf87d3e1649a64c7/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/980f4953e895b7cd1b7ab7eefd0535c7f0213f00a5e71b6e571989e6ea661857/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/006699fbd0e7f47d521943d9a6033ddc895123a489c11ced0897c42b86f751b4/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/5027a6123e4c25c449bfcaf881d465845031dc2af16fca7b52d7cc98bc445335/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/e0cffc8f1a3e4e2fdfb811b58e676373df5162c8b69fafbbd9c74c98f571487b/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/7b25dfa99e278d7f01cae97b62815e97754f63186ba4b524e0386552c003d4cc/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/d15dc925535dd05033f0839f79c08bec2bd58dc745599ffdb663fd377f40d49d/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/155876c62459effdb55284f550041818b6e291e8f877f57850ac45392db4423f/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/76caafbf3a9aa8ac8d69fdb2334f925892c1d828f1e2185d35cd713c9e0f5535/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/f8f9223b5e4ecbd7ba6db1b35e12199d4bdc6c2568be318b5ca54ea492a6c624/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/6b9de7ee6026ad08b833948eecb2c9dbbf31698d7da89d68f00e7f7ed054c7f0/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/0ec384da7f81d6e0e9d93adb296270c0b38bdc5add9c28657e4c133678e9e703/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/f2546223f07b08fabfaa9c493459a74cf2d0307a3e81ea5338a48fab54f8e8e5/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/54a876a3fd2291517f3f1058804935e3ca816e6b5fb849ac510c0ca6f5a2b9af/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/dba2da09295620ca444771a28d24fedfcf22809f6d21133405d24ad8261e1926/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/47621214bd85bfd2a5ba00de6d4912f7099bd587b111910054d6da6e72e74497/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/6efba3c4ef884f6e76f5338e33516f010f2a183cfff05fd1eafcc1d536b7c727/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/4457684b7322613e5e49b4fb282edab97252bf20f8f90144ebe97e2c433b19f0/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/728d136b070407e23a3fc457df1b52c8684d808bd843c5226494de63d2bd76bc/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/8772321d5e2cc3d7be9c3459957b100da4a823efa3f8ba77adc3b72037e3ac13/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/333dae1bbbfa711eb680015e681c66440a432975284c03679440549bf8bd1711/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/ec3c3e006ab29ef173755826e2825f93459f56d3a0d755208e4ad915f5eed326/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/bd6c8c25d111df223f4091a18a9ba060406577979dd3ad862bd54f2a61a196ba/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/e8d0a763ab851e5b4081522166ae11b62424be3420708ec59456fa2dac2f7b76/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/5718817ed5f723055b32334b2593aa395dda6824b8d53bdfc17da4aa94055091/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/9fa567f32a6185c6dc4affef2476eec330e455c7847f618ee95bdf6d638d9e4e/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/7e2350492338e49fa4ec7c5ba373c426850fb68adc33a95fd685689474eb21ce/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/66188315b39ab036ccc42e83e9bda43d12f8c4b7b419d1a1066c6a36f06eae0c/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/bd1fa8cff4f77b06657e7cab91c6f43500634c7f410fdbb6e2916a9a271550d5/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/88d3b816aea30643c898d1e3845105a88162a9f623c8753feef6543a3e82a53e/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/4a4204c8de081b0b3c5c802775adffa011bc3ca26814f06ea6ff91e3b50f38fe/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/2564a19df074ad341d686899222f461cea4f7aac8d9b3c8ec4c2eb6d10d0fd3f/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/f8a485ca9c3cb0323c81e46aa17645534873c091340613f02060f23b7857b1ec/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/b57e4e29342a13e5f4955cbbe8a69167317c1cc3a7ca92baa00c9296bda3f6d8/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/126beffdfcfdb561b58ee830a2b84301624e6d198748edf6d296227f95ac769d/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/bec1e29a4ebee8224e0541d4e2e7944152c10cae8bf16ab579072fc338cd090b/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/d03ac5d1e64ba2a88d68d7e09d05faa968b7dbd1880c28d917d1350823bff320/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/9586739aee92f54af3d2fd9db015221fbbf084d077489e655e86ff8b92c5541c/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/9e75fd6a3c8cab42f713147d902eac9a1037263f94710885161c5c29fe6d0ce1/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/e9e75cc3408561b9210dbcee7216de5de286dd24c0cb378ad15c997344888c3d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/84e5a9ba29b01bcec65fdbc5e01a0764e0295d0565ab0e417417cbf9798e8839/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2919609b9b83b5cf36eadc38f85d16b3ef51a22092d1f6059a55fc76574fb9df/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4fad367e7defc046aed2d5f6d6cf74499be819b3d3b9cc2b104e8bd81ec48993/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ff96282301e64c27c4689fbd37845ead80e4fdae55681804fec86e30db082593/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/00ed127c4dbcaad371ff897ae68af38a819c94dbad33c03f25c4929cc53686ae/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/99e6c57635634d608c3ae4b73b4e3a9ffeb4c0470385ae4d34a09bf34c53fa3f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6e50e0e4530f1d862b83d39a91ed3d0db61ac980abd88d7415070ef3833c65b4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/756fadb35966db8a1a10102bf78ddb9f241fdacbcd151e1aa18d26b74964d159/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8220f33134abd614a0c55071494163a044494b2a5b09b27c63db2a59a461f559/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:17:10.102517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.100414 6514 manager.go:217] Machine: {Timestamp:2026-10-30 19:17:10.099953287 +0000 UTC m=+0.057772468 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117642 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c5517c0a9a2d3769de8f313f2a7f SystemUUID:ec23c551-7c0a-9a2d-3769-de8f313f2a7f BootID:4b717492-2808-4cc9-b740-86026f26beef Filesystems:[{Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a497601e0871e927d1cf5f068f92a5626e19e71c58d7dedec47064e91c38ffaa/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volumes/kubernetes.io~projected/kube-api-access-drbpg DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/471d5763-db1f-46f7-99fe-d55f45fa9a76/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c346bfd4-9367-4a93-a44e-022b444c9f8c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c3588848238282e64b68b07bc36911caeffdfc30b991302f80a42b0c41f554d/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b28eb4af8ae738895dd56130870902cc78727fb6b51b66d35e5675f467a181f0/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:262 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:263 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c4a8b38ed11b915f3d6f1316cabb3f69aa1adb696d57c5d8e1438c7c7f6c598/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0/volumes/kubernetes.io~projected/kube-api-access-tw2pn DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c346bfd4-9367-4a93-a44e-022b444c9f8c/volumes/kubernetes.io~projected/kube-api-access-chvlc DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a565d3d794e5cdfbf73f2db1f50c838059aa78fa9c2b4bb818a20b3f0eec266/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0702df10cc061ca0261fe2ee08defe2b80585a06f7db4e995366ab24029ab62b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:376 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1/userdata/shm DeviceMajor:0 DeviceMinor:425 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26bc4ca2-8079-4560-8891-dc52f796c72f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c9c1b81-6bc0-406b-a0c1-a370afa909b0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:266 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c9c1b81-6bc0-406b-a0c1-a370afa909b0/volumes/kubernetes.io~projected/kube-api-access-m2d7j DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~projected/kube-api-access-7c4cf DeviceMajor:0 DeviceMinor:424 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/618fa036c0be314d1d22b94c16ed2a7557e3d7abbb695c840df9649b6c5ba26a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/166cf5227c8293c52c3eeb4d44d99d72b3b552b1f045c53f363d2f24ddb8eafa/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:423 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/227c72d9c019869b8abcaba1be628512d2a3f3f0e3bc4e968224d7d718c93e9c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:374 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:419 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1174ef776288d115ce0e86a8c5c361a7ef821c2822b1e52a5e914131c717e22/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~projected/kube-api-access-sc85n DeviceMajor:0 DeviceMinor:268 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/862b83b6-5666-4f55-a276-4cb674c0b4a8/volumes/kubernetes.io~projected/kube-api-access-cthhb DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06c453160e25a1f209b81aa3afcb7e7a5c836a7fc4bf6a2b56287ce1d561bfc3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:264 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa4a876-df8d-45b5-aa59-37bf104db694/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3de6224606128cf279e7267cc30d9e24c7152d562234a0f1e12a6c6253df3d21/userdata/shm DeviceMajor:0 DeviceMinor:441 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/53f48b9b-ec52-44f6-a537-e6e585762c7e/volumes/kubernetes.io~projected/kube-api-access-46jgh DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b654842-b7a7-4e8d-8c92-8729111d0373/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:431 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c73bb071e4a02b9ff9a2c54d857849d8f838bfd21fcb69005f1fbbf27f965a7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~projected/kube-api-access-mclx4 DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:267 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f/volumes/kubernetes.io~projected/kube-api-access-th98d DeviceMajor:0 DeviceMinor:437 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e583ac9f2513029b7a03559e9950ed68703dec1df40a56e8d638c047734587c3/userdata/shm DeviceMajor:0 DeviceMinor:440 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad5771d4768d10a5beddd310b0fd405d4db87af384df28d5bb96d5e7060b9fb3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:258 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/15140033-be3b-4729-9a78-3a38b3d1ef48/volumes/kubernetes.io~projected/kube-api-access-v4dn2 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~projected/kube-api-access-4g88x DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:267 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:377 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:267 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:266 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/471d5763-db1f-46f7-99fe-d55f45fa9a76/volumes/kubernetes.io~projected/kube-api-access-tlcq4 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:261 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/06348567-413f-417b-a78b-a10d7ff83ff4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:436 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab/volumes/kubernetes.io~projected/kube-api-access-6qf2c DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/79323c8f-776a-4a3e-942c-a467e8672c53/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/06348567-413f-417b-a78b-a10d7ff83ff4/volumes/kubernetes.io~projected/kube-api-access-9sgj7 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6bdefb407b2d3a49348e70250232bc8762e657306c7e9a64685a70652a5aa73/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41d8bce7eb62c88434d0934b60a6ace44a44edbc1f4a1b51f4233162518298fa/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3489239b-c17e-41ac-84f3-392b94d019e5/volumes/kubernetes.io~projected/kube-api-access-sc9t4 DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da476d5e-0b9c-428d-a53a-98f99d2b15c5/volumes/kubernetes.io~projected/kube-api-access-stb86 DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:267 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ce37e164624398f343e18c717fe837a06a954d765deb6db058b6d119134f3e6/userdata/shm DeviceMajor:0 DeviceMinor:438 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/faee8bf1-c317-4506-a5a7-e7317ab3909d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa4a876-df8d-45b5-aa59-37bf104db694/volumes/kubernetes.io~projected/kube-api-access-l5q52 DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:273 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f438e609-1285-4d67-8622-7b5022bdee31/volumes/kubernetes.io~projected/kube-api-access-8ph5t DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fb8c089-70f4-418d-8cd4-fd2393997ccf/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:272 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes/kubernetes.io~projected/kube-api-access-hn2td DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~projected/kube-api-access-vwnsb DeviceMajor:0 DeviceMinor:337 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ef62b57-79b3-4002-ba56-d41d3b74ded9/volumes/kubernetes.io~projected/kube-api-access-2nhpt DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0f5a9a007a6c0d2bcf736d39e6b96694f04076a7c0c8ca9d337fdcbea800526/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/937e5d34-bca8-4f99-bb4a-22a62c63fe73/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c67ce029859cb7ad74eed0463fc7904bd6caac4b1e4c3969381aed818a42f75a/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03fa645e040b2f5 MacAddress:16:e4:c2:77:28:1b Speed:10000 Mtu:8901} {Name:0702df10cc061ca MacAddress:0e:5f:89:1b:3e:6c Speed:10000 Mtu:8901} {Name:0c4a8b38ed11b91 MacAddress:46:81:cb:b9:4c:ed Speed:10000 Mtu:8901} {Name:1a565d3d794e5cd MacAddress:8a:d2:70:d0:00:6a Speed:10000 Mtu:8901} {Name:2c3588848238282 MacAddress:be:ca:ac:48:26:44 Speed:10000 Mtu:8901} {Name:3de6224606128cf MacAddress:02:92:e7:5f:0b:d7 Speed:10000 Mtu:8901} {Name:4beecbfda0cc805 MacAddress:0e:b1:1d:5a:5a:fa Speed:10000 Mtu:8901} {Name:8ce37e164624398 MacAddress:0e:21:d8:31:32:7e Speed:10000 Mtu:8901} {Name:9c34cdac2db982e MacAddress:1e:94:96:65:32:95 Speed:10000 Mtu:8901} {Name:a497601e0871e92 MacAddress:b6:36:62:fb:0f:05 Speed:10000 Mtu:8901} {Name:b28eb4af8ae7388 MacAddress:7e:20:1d:33:bf:dc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c8:94:42:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:4a:a6:fa:94:11 Speed:0 Mtu:8901} {Name:c67ce029859cb7a MacAddress:8e:e1:b1:1e:2d:b7 Speed:10000 Mtu:8901} {Name:e0f5a9a007a6c0d MacAddress:7a:36:be:d2:66:2f Speed:10000 Mtu:8901} {Name:e583ac9f2513029 MacAddress:b2:b6:e3:ea:02:44 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c8:94:42:81 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:42:23:52:cf:b4:6c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:0d:7c:05:56:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:17:10.102517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.100790 6514 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:17:10.102517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.100828 6514 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:17:10.102517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102099 6514 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:17:10.102949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102154 6514 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-233.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:17:10.102949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102549 6514 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:17:10.102949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102560 6514 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:17:10.102949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102579 6514 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:17:10.102949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102622 6514 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:17:10.102949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.102733 6514 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:17:10.103233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103218 6514 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:17:10.103342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103332 6514 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:17:10.103380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103369 6514 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:17:10.103409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103382 6514 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:17:10.103409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103392 6514 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:17:10.103409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103400 6514 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:17:10.103409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.103407 6514 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:17:10.104221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.104204 6514 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:17:10.104221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.104218 6514 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:17:10.104625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.104609 6514 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:17:10.107484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107468 6514 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:17:10.107484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107482 6514 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:17:10.107587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107568 6514 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:17:10.107741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107730 6514 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:17:10.107775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107759 6514 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:17:10.107844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107836 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107850 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107857 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107882 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107896 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107905 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107915 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107924 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107932 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:17:10.107942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107942 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:17:10.108195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107969 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:17:10.108195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.107984 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:17:10.108195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.108017 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:17:10.108195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.108025 6514 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:17:10.111368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.111353 6514 server.go:1263] "Started kubelet" Oct 30 19:17:10.111452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.111421 6514 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:17:10.114747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.111491 6514 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:17:10.114747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.111677 6514 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:17:10.114747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.112000 6514 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:17:10.114747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.113331 6514 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:17:10.111896 ip-10-0-138-233 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:17:10.119358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.119014 6514 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:17:10.119358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.119030 6514 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:17:10.119358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.119190 6514 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:11:00 +0000 UTC" deadline="2028-04-26 13:16:46.276903873 +0000 UTC" Oct 30 19:17:10.119358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.119218 6514 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13049h59m36.157691536s" Oct 30 19:17:10.119667 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.119377 6514 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-233.us-east-2.compute.internal\" not found" Oct 30 19:17:10.119667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.119452 6514 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:17:10.119667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.119460 6514 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:17:10.121235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.120671 6514 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:17:10.121327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.121260 6514 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:10.123849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123822 6514 factory.go:153] Registering CRI-O factory Oct 30 19:17:10.123960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123852 6514 factory.go:223] Registration of the crio container factory successfully Oct 30 19:17:10.123960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123930 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert" seLinuxMountContext="" Oct 30 19:17:10.123960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123953 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-client-ca-bundle" seLinuxMountContext="" Oct 30 19:17:10.123960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123962 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123969 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53f48b9b-ec52-44f6-a537-e6e585762c7e" volumeName="kubernetes.io/projected/53f48b9b-ec52-44f6-a537-e6e585762c7e-kube-api-access-46jgh" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123977 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb8c089-70f4-418d-8cd4-fd2393997ccf" volumeName="kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123984 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79323c8f-776a-4a3e-942c-a467e8672c53" volumeName="kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-env-overrides" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.123994 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79323c8f-776a-4a3e-942c-a467e8672c53" volumeName="kubernetes.io/secret/79323c8f-776a-4a3e-942c-a467e8672c53-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124001 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/empty-dir/937e5d34-bca8-4f99-bb4a-22a62c63fe73-audit-log" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124003 6514 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124010 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124014 6514 factory.go:55] Registering systemd factory Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124017 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f" volumeName="kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124025 6514 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124028 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124036 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15140033-be3b-4729-9a78-3a38b3d1ef48" volumeName="kubernetes.io/configmap/15140033-be3b-4729-9a78-3a38b3d1ef48-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124044 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124051 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb8c089-70f4-418d-8cd4-fd2393997ccf" volumeName="kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124073 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79323c8f-776a-4a3e-942c-a467e8672c53" volumeName="kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-config" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124080 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="862b83b6-5666-4f55-a276-4cb674c0b4a8" volumeName="kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-daemon-config" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124087 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124094 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124102 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471d5763-db1f-46f7-99fe-d55f45fa9a76" volumeName="kubernetes.io/projected/471d5763-db1f-46f7-99fe-d55f45fa9a76-kube-api-access-tlcq4" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124108 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef62b57-79b3-4002-ba56-d41d3b74ded9" volumeName="kubernetes.io/projected/9ef62b57-79b3-4002-ba56-d41d3b74ded9-kube-api-access-2nhpt" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124115 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124122 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124129 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124136 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faee8bf1-c317-4506-a5a7-e7317ab3909d" volumeName="kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-tls" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124143 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124150 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124161 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faee8bf1-c317-4506-a5a7-e7317ab3909d" volumeName="kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124174 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3489239b-c17e-41ac-84f3-392b94d019e5" volumeName="kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-tmp" seLinuxMountContext="" Oct 30 19:17:10.124166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124184 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b654842-b7a7-4e8d-8c92-8729111d0373" volumeName="kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124191 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124209 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9c1b81-6bc0-406b-a0c1-a370afa909b0" volumeName="kubernetes.io/projected/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-kube-api-access-m2d7j" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124219 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9c1b81-6bc0-406b-a0c1-a370afa909b0" volumeName="kubernetes.io/secret/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-metrics-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124240 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124262 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb8c089-70f4-418d-8cd4-fd2393997ccf" volumeName="kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124268 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79323c8f-776a-4a3e-942c-a467e8672c53" volumeName="kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124276 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c346bfd4-9367-4a93-a44e-022b444c9f8c" volumeName="kubernetes.io/empty-dir/c346bfd4-9367-4a93-a44e-022b444c9f8c-data-volume" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124290 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c346bfd4-9367-4a93-a44e-022b444c9f8c" volumeName="kubernetes.io/secret/c346bfd4-9367-4a93-a44e-022b444c9f8c-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124303 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faee8bf1-c317-4506-a5a7-e7317ab3909d" volumeName="kubernetes.io/projected/faee8bf1-c317-4506-a5a7-e7317ab3909d-kube-api-access-mclx4" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124314 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef62b57-79b3-4002-ba56-d41d3b74ded9" volumeName="kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124325 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124349 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3489239b-c17e-41ac-84f3-392b94d019e5" volumeName="kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-etc-tuned" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124361 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9c1b81-6bc0-406b-a0c1-a370afa909b0" volumeName="kubernetes.io/configmap/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-config-volume" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124374 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124384 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124395 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da476d5e-0b9c-428d-a53a-98f99d2b15c5" volumeName="kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124406 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da476d5e-0b9c-428d-a53a-98f99d2b15c5" volumeName="kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124417 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/projected/f438e609-1285-4d67-8622-7b5022bdee31-kube-api-access-8ph5t" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124428 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15140033-be3b-4729-9a78-3a38b3d1ef48" volumeName="kubernetes.io/projected/15140033-be3b-4729-9a78-3a38b3d1ef48-kube-api-access-v4dn2" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124439 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3489239b-c17e-41ac-84f3-392b94d019e5" volumeName="kubernetes.io/projected/3489239b-c17e-41ac-84f3-392b94d019e5-kube-api-access-sc9t4" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124450 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb8c089-70f4-418d-8cd4-fd2393997ccf" volumeName="kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124461 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124473 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c346bfd4-9367-4a93-a44e-022b444c9f8c" volumeName="kubernetes.io/projected/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-api-access-chvlc" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124490 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124506 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124522 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06348567-413f-417b-a78b-a10d7ff83ff4" volumeName="kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124534 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab" volumeName="kubernetes.io/empty-dir/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-tmp-dir" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124559 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26bc4ca2-8079-4560-8891-dc52f796c72f" volumeName="kubernetes.io/configmap/26bc4ca2-8079-4560-8891-dc52f796c72f-konnectivity-ca" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124571 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9c1b81-6bc0-406b-a0c1-a370afa909b0" volumeName="kubernetes.io/empty-dir/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-tmp-dir" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124584 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124595 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa4a876-df8d-45b5-aa59-37bf104db694" volumeName="kubernetes.io/projected/bfa4a876-df8d-45b5-aa59-37bf104db694-kube-api-access-l5q52" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124607 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124619 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da476d5e-0b9c-428d-a53a-98f99d2b15c5" volumeName="kubernetes.io/projected/da476d5e-0b9c-428d-a53a-98f99d2b15c5-kube-api-access-stb86" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124625 6514 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124629 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06348567-413f-417b-a78b-a10d7ff83ff4" volumeName="kubernetes.io/projected/06348567-413f-417b-a78b-a10d7ff83ff4-kube-api-access-9sgj7" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124673 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb8c089-70f4-418d-8cd4-fd2393997ccf" volumeName="kubernetes.io/projected/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-api-access-4g88x" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124685 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124697 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124711 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124722 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faee8bf1-c317-4506-a5a7-e7317ab3909d" volumeName="kubernetes.io/empty-dir/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-textfile" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124734 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df29a6a-001e-463a-9a07-0cfe1c0fe6a0" volumeName="kubernetes.io/projected/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-kube-api-access-tw2pn" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124746 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79323c8f-776a-4a3e-942c-a467e8672c53" volumeName="kubernetes.io/projected/79323c8f-776a-4a3e-942c-a467e8672c53-kube-api-access-drbpg" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124760 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="862b83b6-5666-4f55-a276-4cb674c0b4a8" volumeName="kubernetes.io/projected/862b83b6-5666-4f55-a276-4cb674c0b4a8-kube-api-access-cthhb" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124772 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124784 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124795 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124806 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da476d5e-0b9c-428d-a53a-98f99d2b15c5" volumeName="kubernetes.io/configmap/da476d5e-0b9c-428d-a53a-98f99d2b15c5-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124817 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c346bfd4-9367-4a93-a44e-022b444c9f8c" volumeName="kubernetes.io/configmap/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124828 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124840 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124851 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-client-tls" seLinuxMountContext="" Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124046 6514 factory.go:103] Registering Raw factory Oct 30 19:17:10.125061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124902 6514 manager.go:1199] Started watching for new ooms in manager Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125416 6514 manager.go:319] Starting recovery of all containers Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.124877 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faee8bf1-c317-4506-a5a7-e7317ab3909d" volumeName="kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125738 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125758 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125773 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb8c089-70f4-418d-8cd4-fd2393997ccf" volumeName="kubernetes.io/empty-dir/6fb8c089-70f4-418d-8cd4-fd2393997ccf-volume-directive-shadow" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125786 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="862b83b6-5666-4f55-a276-4cb674c0b4a8" volumeName="kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-cni-binary-copy" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125800 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/projected/937e5d34-bca8-4f99-bb4a-22a62c63fe73-kube-api-access-vwnsb" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125812 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937e5d34-bca8-4f99-bb4a-22a62c63fe73" volumeName="kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125831 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef62b57-79b3-4002-ba56-d41d3b74ded9" volumeName="kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125846 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" volumeName="kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125857 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471d5763-db1f-46f7-99fe-d55f45fa9a76" volumeName="kubernetes.io/secret/471d5763-db1f-46f7-99fe-d55f45fa9a76-cert" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125884 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef62b57-79b3-4002-ba56-d41d3b74ded9" volumeName="kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-binary-copy" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125899 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125911 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa4a876-df8d-45b5-aa59-37bf104db694" volumeName="kubernetes.io/secret/bfa4a876-df8d-45b5-aa59-37bf104db694-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125926 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f438e609-1285-4d67-8622-7b5022bdee31" volumeName="kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-federate-client-tls" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125939 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faee8bf1-c317-4506-a5a7-e7317ab3909d" volumeName="kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125951 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d0014b-03d6-4271-8b1b-3749435c511f" volumeName="kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125963 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab" volumeName="kubernetes.io/projected/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-kube-api-access-6qf2c" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125981 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df29a6a-001e-463a-9a07-0cfe1c0fe6a0" volumeName="kubernetes.io/configmap/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-serviceca" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.125994 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26bc4ca2-8079-4560-8891-dc52f796c72f" volumeName="kubernetes.io/secret/26bc4ca2-8079-4560-8891-dc52f796c72f-agent-certs" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.126009 6514 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" volumeName="kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config" seLinuxMountContext="" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.126021 6514 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.126029 6514 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:17:10.126543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.126241 6514 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:10.130062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.130041 6514 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:17:10.145097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.145072 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:17:10.145665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.145633 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:17:10.149443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.149414 6514 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:17:10.150783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.150761 6514 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:17:10.150783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.150784 6514 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:17:10.150942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.150809 6514 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:17:10.150942 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.150857 6514 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:17:10.155918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.155901 6514 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:10.166157 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.166140 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:17:10.166535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.166507 6514 generic.go:356] "Generic (PLEG): container finished" podID="79323c8f-776a-4a3e-942c-a467e8672c53" containerID="45872986aa95c74662be8ea04a9ddf2b175c5992556ac9b5a77deb536bc845e1" exitCode=1 Oct 30 19:17:10.169958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.169936 6514 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="4d6ed7bff4e7d31f7a6eebb5e39f29c351337fe4f6e523553a3a14781fbab91d" exitCode=0 Oct 30 19:17:10.169958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.169957 6514 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="76cef0201b34454b210a1ec92af1a7e2283e46adc0395a65e7cad5e0365d01cc" exitCode=0 Oct 30 19:17:10.170054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.169964 6514 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="b21a68be9b27e2a3e1818ddd35952d4bea2ac5cb7734e58ae8cc06626948d3d8" exitCode=0 Oct 30 19:17:10.170054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.169969 6514 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="3dd302ad56ebd9720f8eed8b6a3db81ab2b21bab37c4d0553ec4ed2a6e24e9bf" exitCode=0 Oct 30 19:17:10.170054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.169974 6514 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="eca04dd31615fb71f75f7561c8a61f21574250ee015ae77a10665d82958e8551" exitCode=0 Oct 30 19:17:10.170054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.169981 6514 generic.go:356] "Generic (PLEG): container finished" podID="9ef62b57-79b3-4002-ba56-d41d3b74ded9" containerID="52aaa5f8574c158d5763046b1a8a6d64f35f35eab0b739e47c87ba677a6fcf50" exitCode=0 Oct 30 19:17:10.198235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.198211 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee" exitCode=0 Oct 30 19:17:10.202568 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.202544 6514 generic.go:356] "Generic (PLEG): container finished" podID="faee8bf1-c317-4506-a5a7-e7317ab3909d" containerID="b717c7fd979f0b5a50fa1fc1a329f7c0245d1f13219d50485ab9db76c9044bab" exitCode=0 Oct 30 19:17:10.209729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.209509 6514 generic.go:356] "Generic (PLEG): container finished" podID="b0906d8be50774c4e8328161c8119d16" containerID="01f2d98a242cdce7216d71858868589efc34721cead3d7075f93edd5386d5205" exitCode=0 Oct 30 19:17:10.234958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.234943 6514 manager.go:324] Recovery completed Oct 30 19:17:10.251501 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.251473 6514 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 19:17:10.258448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258400 6514 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:17:10.258448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258418 6514 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:17:10.258448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258439 6514 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:17:10.258576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258563 6514 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:17:10.258605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258575 6514 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:17:10.258605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258591 6514 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:17:10.258605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258596 6514 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:17:10.258605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258601 6514 policy_none.go:50] "Start" Oct 30 19:17:10.258710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258612 6514 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:17:10.258710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258621 6514 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:17:10.258759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258728 6514 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:17:10.258759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258735 6514 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:17:10.258759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.258740 6514 policy_none.go:44] "Start" Oct 30 19:17:10.263226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.263203 6514 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:17:10.268747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.268733 6514 manager.go:341] "Starting Device Plugin manager" Oct 30 19:17:10.269199 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.268761 6514 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:17:10.269271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269202 6514 server.go:87] "Starting device plugin registration server" Oct 30 19:17:10.269483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269469 6514 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:17:10.269557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269483 6514 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:17:10.269608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269559 6514 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:17:10.269769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269668 6514 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:17:10.269769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269724 6514 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:17:10.269769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269730 6514 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:17:10.269952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.269776 6514 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:17:10.269709115Z","UUID":"f1d41682-3b20-4447-9e84-ecd3c89acec1","Handler":null,"Name":"","Endpoint":""} Oct 30 19:17:10.271507 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.271487 6514 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:17:10.272364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.272344 6514 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:17:10.272449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.272381 6514 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:17:10.370519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.370498 6514 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:17:10.372425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.372408 6514 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:17:10.372502 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.372443 6514 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:17:10.372502 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.372459 6514 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-233.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:17:10.372605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.372515 6514 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.400502 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.400483 6514 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.400564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.400533 6514 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.452342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.452308 6514 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal"] Oct 30 19:17:10.452942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.452927 6514 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.453007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.452980 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"0bda5fbf6e9e2678ca515373c50ec8ae99e8355eaf78dc416d893d716a40523b"} Oct 30 19:17:10.453075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.453022 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"2cb181debb656c62bb70f1078166094d","Type":"ContainerStarted","Data":"26373d785dedaa97da467da71d30fdd43b9adf6df534985cd98419b60ef41557"} Oct 30 19:17:10.453128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.453084 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"e89b039a1f76e192410796bf6b847693a250491066b525cffab0c527ee63cb56"} Oct 30 19:17:10.453128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.453106 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerDied","Data":"01f2d98a242cdce7216d71858868589efc34721cead3d7075f93edd5386d5205"} Oct 30 19:17:10.453128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.453125 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" event={"ID":"b0906d8be50774c4e8328161c8119d16","Type":"ContainerStarted","Data":"0f5461dfe2ad419e5e43c5a72150ac82b5e8669d88263323d45366d97436177a"} Oct 30 19:17:10.453269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.453167 6514 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.468202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.468169 6514 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:10.468314 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.468211 6514 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.469296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.469279 6514 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:10.469376 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:10.469310 6514 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.531636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.531584 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.531636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.531610 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.531636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.531624 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.632151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.632134 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.632197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.632156 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.632197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.632171 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.632272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.632213 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.632272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.632228 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0906d8be50774c4e8328161c8119d16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"b0906d8be50774c4e8328161c8119d16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:10.632272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:10.632213 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cb181debb656c62bb70f1078166094d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" (UID: \"2cb181debb656c62bb70f1078166094d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:11.103982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.103958 6514 apiserver.go:52] "Watching apiserver" Oct 30 19:17:11.122931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.122912 6514 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:17:11.123610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.123584 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mkrdd","openshift-monitoring/metrics-server-8988ff5d4-t8x89","openshift-multus/multus-w2lmc","openshift-dns/node-resolver-p74tq","openshift-network-diagnostics/network-check-target-tfj5g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-additional-cni-plugins-xbzbq","openshift-multus/network-metrics-daemon-lc88s","kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-ddx75","openshift-console/console-56c4467959-2ccth","openshift-insights/insights-runtime-extractor-54p6s","openshift-monitoring/monitoring-plugin-77f4f76594-slbxc","openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x","openshift-ingress-canary/ingress-canary-bpk2j","openshift-network-operator/iptables-alerter-mwhv4","openshift-console/console-5d5d44cd94-pfbqm","openshift-ovn-kubernetes/ovnkube-node-p682z","openshift-dns/dns-default-fjmnw","kube-system/konnectivity-agent-9ssdd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv","openshift-monitoring/node-exporter-ftfsr","openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf","openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz","kube-system/global-pull-secret-syncer-5sqtd"] Oct 30 19:17:11.130492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.130173 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9tgzd\"" Oct 30 19:17:11.130492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.130233 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:17:11.130492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.130309 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.130492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.130364 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrdg\"" Oct 30 19:17:11.130492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.130362 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qmpzj\"" Oct 30 19:17:11.130492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.130384 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.131324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131304 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:17:11.131324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131329 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.131468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131411 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.131468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131444 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.131574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131510 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:17:11.131574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131534 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.131574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131556 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:17:11.131574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131567 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.131737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131566 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.131737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131655 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6xn8\"" Oct 30 19:17:11.131948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.131930 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.133180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133163 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzjv5\"" Oct 30 19:17:11.133311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133295 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:17:11.133418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133370 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.133418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133387 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:17:11.133418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133411 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:17:11.133522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133410 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.133522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133519 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:17:11.133613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133550 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.133613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133599 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\"" Oct 30 19:17:11.133714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133678 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.133768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133749 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:17:11.133823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133806 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:17:11.133976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.133962 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7s42\"" Oct 30 19:17:11.134034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.134005 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:17:11.134072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.134054 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mglp5\"" Oct 30 19:17:11.135098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135077 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xph5z\"" Oct 30 19:17:11.135295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135122 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.135295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135154 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.135543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135395 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.135543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135468 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:17:11.135543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135501 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:17:11.135543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135520 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\"" Oct 30 19:17:11.135543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135543 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wctlt\"" Oct 30 19:17:11.135888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.135657 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:17:11.136972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.136943 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:17:11.137064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.136978 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.137064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137037 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:17:11.137286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137066 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:17:11.137352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137316 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.137352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137335 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.137352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137342 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:17:11.137491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137355 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.137491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137335 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bfjrl\"" Oct 30 19:17:11.137491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137410 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\"" Oct 30 19:17:11.137818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.137800 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7b7kv\"" Oct 30 19:17:11.138176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138161 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:17:11.138805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138783 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:17:11.138906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138848 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.138906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138849 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:17:11.138906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138857 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bc6qp\"" Oct 30 19:17:11.138906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138907 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jzpj7\"" Oct 30 19:17:11.139238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.138936 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:17:11.139333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.139323 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:17:11.139407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.139390 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:17:11.139507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.139431 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfmhh\"" Oct 30 19:17:11.139507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.139452 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.139609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.139514 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:17:11.139706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.139691 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:17:11.140698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.140679 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:17:11.140784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.140732 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:17:11.140784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.140735 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:17:11.140896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.140806 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jbwq9\"" Oct 30 19:17:11.140896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.140843 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141233 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141243 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141263 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141262 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141279 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hlrws\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141264 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:17:11.141293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.141284 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:17:11.142969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.142952 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:17:11.146933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.146909 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9vapspveagene\"" Oct 30 19:17:11.150223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.147013 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:17:11.150298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.150220 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:17:11.217012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.216996 6514 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:11.217121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.217109 6514 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:11.223397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.223381 6514 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:17:11.229389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229375 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:17:11.229600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229577 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nxtgr\"" Oct 30 19:17:11.229671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229622 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:17:11.229671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229645 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:17:11.229742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229677 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 19:17:11.229927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229913 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w7ctm\"" Oct 30 19:17:11.229972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229925 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:17:11.229972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229934 6514 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:11.229972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.229925 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:17:11.230067 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:11.229989 6514 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:11.230067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230050 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 19:17:11.230129 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230101 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:17:11.230199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230185 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:17:11.230199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230195 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:17:11.230258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230206 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:17:11.230258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230212 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 19:17:11.230258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230229 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:17:11.230258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230243 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:17:11.230709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230698 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 19:17:11.230746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230708 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-78j2g\"" Oct 30 19:17:11.230851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.230841 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 19:17:11.231476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.231461 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n75qq\"" Oct 30 19:17:11.234020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234002 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.234101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234025 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.234101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234045 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-sys-fs\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.234101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234060 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-cnibin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.234101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234100 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-bin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.234229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234119 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclx4\" (UniqueName: \"kubernetes.io/projected/faee8bf1-c317-4506-a5a7-e7317ab3909d-kube-api-access-mclx4\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.234229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234179 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-metrics-client-ca\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.234290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234284 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.234353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234336 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-socket-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.234401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234383 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fb8c089-70f4-418d-8cd4-fd2393997ccf-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.234452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234415 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.234452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234418 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.234452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234439 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6fb8c089-70f4-418d-8cd4-fd2393997ccf-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.234546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234451 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-host\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.234546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234474 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-k8s-cni-cncf-io\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.234546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234500 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-kubelet\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.234546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234510 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-metrics-client-ca\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.234546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234535 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.234724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234563 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-node-log\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.234724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234581 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-root\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.234724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234606 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.234724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234712 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.234876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234755 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.234876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234775 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q52\" (UniqueName: \"kubernetes.io/projected/bfa4a876-df8d-45b5-aa59-37bf104db694-kube-api-access-l5q52\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:17:11.234876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234808 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stb86\" (UniqueName: \"kubernetes.io/projected/da476d5e-0b9c-428d-a53a-98f99d2b15c5-kube-api-access-stb86\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.234876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234811 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.234876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234842 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-textfile\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.235093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234892 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-dbus\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.235093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.234950 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-textfile\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.235093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235014 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-netns\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.235093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235070 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-hostroot\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.235215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235109 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/937e5d34-bca8-4f99-bb4a-22a62c63fe73-audit-log\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.235215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235151 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-var-lib-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.235215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235202 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-ovn\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.235215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235204 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/937e5d34-bca8-4f99-bb4a-22a62c63fe73-audit-log\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235233 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysconfig\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235258 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-metrics-client-ca\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235300 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235320 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235335 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235352 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-lib-modules\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.235375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235369 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhpt\" (UniqueName: \"kubernetes.io/projected/9ef62b57-79b3-4002-ba56-d41d3b74ded9-kube-api-access-2nhpt\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235386 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235398 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235412 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-bin\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235452 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-metrics-client-ca\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235487 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-script-lib\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235538 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ph5t\" (UniqueName: \"kubernetes.io/projected/f438e609-1285-4d67-8622-7b5022bdee31-kube-api-access-8ph5t\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235571 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-kubernetes\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235542 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235605 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-system-cni-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.235678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235679 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235738 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-script-lib\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235783 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235811 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b654842-b7a7-4e8d-8c92-8729111d0373-original-pull-secret\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235820 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-client-certs\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235847 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwnsb\" (UniqueName: \"kubernetes.io/projected/937e5d34-bca8-4f99-bb4a-22a62c63fe73-kube-api-access-vwnsb\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235890 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235907 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235913 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-etc-kubernetes\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.235966 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-slash\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236015 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236064 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-netd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236091 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-client-certs\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236101 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26bc4ca2-8079-4560-8891-dc52f796c72f-konnectivity-ca\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:17:11.236159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236150 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-run\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236180 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c346bfd4-9367-4a93-a44e-022b444c9f8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236203 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-device-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236352 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26bc4ca2-8079-4560-8891-dc52f796c72f-konnectivity-ca\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236370 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236392 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-daemon-config\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236417 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236433 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-systemd-units\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236494 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c346bfd4-9367-4a93-a44e-022b444c9f8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236520 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-metrics-tls\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236544 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-tmp-dir\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236563 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.236590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236591 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236606 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236613 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-tmp-dir\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236617 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236567 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-daemon-config\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236654 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-wtmp\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236709 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236726 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236737 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236737 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236746 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-metrics-tls\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236779 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jgh\" (UniqueName: \"kubernetes.io/projected/53f48b9b-ec52-44f6-a537-e6e585762c7e-kube-api-access-46jgh\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236798 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-multus-certs\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236918 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236939 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236947 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79323c8f-776a-4a3e-942c-a467e8672c53-ovn-node-metrics-cert\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.236979 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237001 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9t4\" (UniqueName: \"kubernetes.io/projected/3489239b-c17e-41ac-84f3-392b94d019e5-kube-api-access-sc9t4\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237043 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-kubelet\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237067 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcq4\" (UniqueName: \"kubernetes.io/projected/471d5763-db1f-46f7-99fe-d55f45fa9a76-kube-api-access-tlcq4\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237079 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237092 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:11.237107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237117 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237166 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79323c8f-776a-4a3e-942c-a467e8672c53-ovn-node-metrics-cert\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237177 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-tls\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237190 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-secret-telemeter-client\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237252 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-var-lib-kubelet\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237280 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237358 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237442 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-multus\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237471 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-secret-metrics-server-tls\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237491 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbpg\" (UniqueName: \"kubernetes.io/projected/79323c8f-776a-4a3e-942c-a467e8672c53-kube-api-access-drbpg\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237512 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-hosts-file\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237527 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-modprobe-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237541 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-etc-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237557 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237591 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237609 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cnibin\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237640 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15140033-be3b-4729-9a78-3a38b3d1ef48-host-slash\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237667 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-registration-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237690 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237727 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-serving-certs-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237801 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfa4a876-df8d-45b5-aa59-37bf104db694-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:17:11.237820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237825 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-etc-tuned\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237843 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237857 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237899 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-kubelet-config\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237923 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthhb\" (UniqueName: \"kubernetes.io/projected/862b83b6-5666-4f55-a276-4cb674c0b4a8-kube-api-access-cthhb\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237937 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-serving-certs-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237944 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d5763-db1f-46f7-99fe-d55f45fa9a76-cert\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237939 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-etc-tuned\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.237968 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvlc\" (UniqueName: \"kubernetes.io/projected/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-api-access-chvlc\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238011 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238059 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-config-volume\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238097 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-env-overrides\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238122 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-sys\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238166 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238176 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfa4a876-df8d-45b5-aa59-37bf104db694-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238192 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238196 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238296 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d5763-db1f-46f7-99fe-d55f45fa9a76-cert\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238346 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-env-overrides\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238345 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-conf-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238416 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238464 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238495 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-federate-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.238529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238511 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238617 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-host\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238649 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-config\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238662 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-config-volume\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.238972 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79323c8f-776a-4a3e-942c-a467e8672c53-ovnkube-config\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239145 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239185 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239213 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239240 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239264 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239288 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239315 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-serviceca\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239363 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2pn\" (UniqueName: \"kubernetes.io/projected/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-kube-api-access-tw2pn\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239391 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-client-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239416 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239439 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239439 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2d7j\" (UniqueName: \"kubernetes.io/projected/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-kube-api-access-m2d7j\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.239560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239486 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239610 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239639 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239675 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g88x\" (UniqueName: \"kubernetes.io/projected/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-api-access-4g88x\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239695 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239889 6514 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:11.239920 6514 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.us-east-2.compute.internal" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239923 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.239988 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240068 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/937e5d34-bca8-4f99-bb4a-22a62c63fe73-client-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240302 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.240366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240323 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-serviceca\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240449 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240535 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-log-socket\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240538 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06348567-413f-417b-a78b-a10d7ff83ff4-metrics-certs\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240590 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgj7\" (UniqueName: \"kubernetes.io/projected/06348567-413f-417b-a78b-a10d7ff83ff4-kube-api-access-9sgj7\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240627 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-systemd\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240669 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15140033-be3b-4729-9a78-3a38b3d1ef48-iptables-alerter-script\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240788 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240813 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.240845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240817 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-systemd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240857 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-federate-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240894 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26bc4ca2-8079-4560-8891-dc52f796c72f-agent-certs\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240925 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240928 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15140033-be3b-4729-9a78-3a38b3d1ef48-iptables-alerter-script\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.240951 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-system-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241051 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241091 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241154 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26bc4ca2-8079-4560-8891-dc52f796c72f-agent-certs\") pod \"konnectivity-agent-9ssdd\" (UID: \"26bc4ca2-8079-4560-8891-dc52f796c72f\") " pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241181 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-client-tls\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241190 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-cni-binary-copy\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241232 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.241270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241262 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241287 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241347 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241375 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241399 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241424 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-netns\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241450 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qf2c\" (UniqueName: \"kubernetes.io/projected/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-kube-api-access-6qf2c\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241475 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-sys\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241709 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241739 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241766 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241790 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-conf\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.241815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241817 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c346bfd4-9367-4a93-a44e-022b444c9f8c-crio-socket\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241842 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da476d5e-0b9c-428d-a53a-98f99d2b15c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241885 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.241910 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242072 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242085 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242190 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242193 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.242319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242248 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862b83b6-5666-4f55-a276-4cb674c0b4a8-cni-binary-copy\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.242544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242338 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da476d5e-0b9c-428d-a53a-98f99d2b15c5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.242544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242370 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.242544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242414 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cni-binary-copy\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.242544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242477 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.242544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242505 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.242544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242506 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242545 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-socket-dir-parent\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242573 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242601 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-metrics-server-audit-profiles\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242625 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-tmp-dir\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242650 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-tmp\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242673 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-tls\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242669 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242703 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dn2\" (UniqueName: \"kubernetes.io/projected/15140033-be3b-4729-9a78-3a38b3d1ef48-kube-api-access-v4dn2\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.242743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242648 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242772 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c346bfd4-9367-4a93-a44e-022b444c9f8c-data-volume\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242802 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242821 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3489239b-c17e-41ac-84f3-392b94d019e5-tmp\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242884 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-tls\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242931 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-tmp-dir\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.242949 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c346bfd4-9367-4a93-a44e-022b444c9f8c-data-volume\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.243081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.243066 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/937e5d34-bca8-4f99-bb4a-22a62c63fe73-metrics-server-audit-profiles\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.243282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.243131 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da476d5e-0b9c-428d-a53a-98f99d2b15c5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.243956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.243937 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclx4\" (UniqueName: \"kubernetes.io/projected/faee8bf1-c317-4506-a5a7-e7317ab3909d-kube-api-access-mclx4\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.245109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.245087 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q52\" (UniqueName: \"kubernetes.io/projected/bfa4a876-df8d-45b5-aa59-37bf104db694-kube-api-access-l5q52\") pod \"monitoring-plugin-77f4f76594-slbxc\" (UID: \"bfa4a876-df8d-45b5-aa59-37bf104db694\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:17:11.245435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.245417 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbpg\" (UniqueName: \"kubernetes.io/projected/79323c8f-776a-4a3e-942c-a467e8672c53-kube-api-access-drbpg\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.245631 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.245609 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ph5t\" (UniqueName: \"kubernetes.io/projected/f438e609-1285-4d67-8622-7b5022bdee31-kube-api-access-8ph5t\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.245970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.245949 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthhb\" (UniqueName: \"kubernetes.io/projected/862b83b6-5666-4f55-a276-4cb674c0b4a8-kube-api-access-cthhb\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.246479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.246286 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f438e609-1285-4d67-8622-7b5022bdee31-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78bcbbb9fb-frn5x\" (UID: \"f438e609-1285-4d67-8622-7b5022bdee31\") " pod="openshift-monitoring/telemeter-client-78bcbbb9fb-frn5x" Oct 30 19:17:11.248180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.247514 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jgh\" (UniqueName: \"kubernetes.io/projected/53f48b9b-ec52-44f6-a537-e6e585762c7e-kube-api-access-46jgh\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.248180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.247736 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcq4\" (UniqueName: \"kubernetes.io/projected/471d5763-db1f-46f7-99fe-d55f45fa9a76-kube-api-access-tlcq4\") pod \"ingress-canary-bpk2j\" (UID: \"471d5763-db1f-46f7-99fe-d55f45fa9a76\") " pod="openshift-ingress-canary/ingress-canary-bpk2j" Oct 30 19:17:11.248180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.247737 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwnsb\" (UniqueName: \"kubernetes.io/projected/937e5d34-bca8-4f99-bb4a-22a62c63fe73-kube-api-access-vwnsb\") pod \"metrics-server-8988ff5d4-t8x89\" (UID: \"937e5d34-bca8-4f99-bb4a-22a62c63fe73\") " pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:11.248180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.247779 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9t4\" (UniqueName: \"kubernetes.io/projected/3489239b-c17e-41ac-84f3-392b94d019e5-kube-api-access-sc9t4\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.248180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.248109 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvlc\" (UniqueName: \"kubernetes.io/projected/c346bfd4-9367-4a93-a44e-022b444c9f8c-kube-api-access-chvlc\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.248469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.248293 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stb86\" (UniqueName: \"kubernetes.io/projected/da476d5e-0b9c-428d-a53a-98f99d2b15c5-kube-api-access-stb86\") pod \"openshift-state-metrics-7bd7489957-xfsnz\" (UID: \"da476d5e-0b9c-428d-a53a-98f99d2b15c5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xfsnz" Oct 30 19:17:11.248469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.248339 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhpt\" (UniqueName: \"kubernetes.io/projected/9ef62b57-79b3-4002-ba56-d41d3b74ded9-kube-api-access-2nhpt\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.248469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.248434 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") pod \"console-5d5d44cd94-pfbqm\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:11.248607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.248517 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th98d\" (UniqueName: \"kubernetes.io/projected/e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f-kube-api-access-th98d\") pod \"network-check-target-tfj5g\" (UID: \"e00ccf1e-da25-4bd2-ae4c-eb09fc8c0d6f\") " pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:11.250505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.249912 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g88x\" (UniqueName: \"kubernetes.io/projected/6fb8c089-70f4-418d-8cd4-fd2393997ccf-kube-api-access-4g88x\") pod \"kube-state-metrics-5dc8d7d89b-277gf\" (UID: \"6fb8c089-70f4-418d-8cd4-fd2393997ccf\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-277gf" Oct 30 19:17:11.250505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.250079 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2pn\" (UniqueName: \"kubernetes.io/projected/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-kube-api-access-tw2pn\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.250505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.250165 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2d7j\" (UniqueName: \"kubernetes.io/projected/3c9c1b81-6bc0-406b-a0c1-a370afa909b0-kube-api-access-m2d7j\") pod \"dns-default-fjmnw\" (UID: \"3c9c1b81-6bc0-406b-a0c1-a370afa909b0\") " pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:11.250505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.250479 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgj7\" (UniqueName: \"kubernetes.io/projected/06348567-413f-417b-a78b-a10d7ff83ff4-kube-api-access-9sgj7\") pod \"network-metrics-daemon-lc88s\" (UID: \"06348567-413f-417b-a78b-a10d7ff83ff4\") " pod="openshift-multus/network-metrics-daemon-lc88s" Oct 30 19:17:11.250743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.250604 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dn2\" (UniqueName: \"kubernetes.io/projected/15140033-be3b-4729-9a78-3a38b3d1ef48-kube-api-access-v4dn2\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.250792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.250747 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qf2c\" (UniqueName: \"kubernetes.io/projected/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-kube-api-access-6qf2c\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.251592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.250951 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") pod \"alertmanager-main-0\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:11.251592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.251338 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") pod \"console-56c4467959-2ccth\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:11.343449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343426 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-sys-fs\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.343547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343455 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-cnibin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343470 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-bin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343485 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-socket-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343547 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-cnibin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343557 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-socket-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343562 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-bin\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343557 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-sys-fs\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343589 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343595 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343616 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-host\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343632 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-k8s-cni-cncf-io\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343649 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1df29a6a-001e-463a-9a07-0cfe1c0fe6a0-host\") pod \"node-ca-mkrdd\" (UID: \"1df29a6a-001e-463a-9a07-0cfe1c0fe6a0\") " pod="openshift-image-registry/node-ca-mkrdd" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343655 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-kubelet\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343681 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-node-log\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343684 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-k8s-cni-cncf-io\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343704 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-root\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343684 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-kubelet\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343726 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343750 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-dbus\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343751 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343771 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-root\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343780 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-node-log\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343808 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-netns\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343813 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-dbus\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343832 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-hostroot\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343852 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-var-lib-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343886 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-ovn\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343852 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-netns\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.343912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343910 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysconfig\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343926 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-var-lib-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343948 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-ovn\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343956 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-hostroot\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343971 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysconfig\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343980 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-lib-modules\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.343996 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-bin\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344012 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-kubernetes\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344026 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-system-cni-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344048 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-etc-kubernetes\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344062 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-slash\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344080 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-lib-modules\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344089 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-bin\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344096 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-slash\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344107 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-netd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344124 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-etc-kubernetes\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344132 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-run\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344127 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-system-cni-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344135 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-kubernetes\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344149 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-device-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344166 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-cni-netd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344168 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-systemd-units\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344178 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-run\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344209 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-device-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344212 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-systemd-units\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344233 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-wtmp\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344261 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344276 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-multus-certs\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344291 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-kubelet\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344306 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-var-lib-kubelet\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344319 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-multus\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344333 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-hosts-file\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344341 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-node-exporter-wtmp\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344364 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-kubelet\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344369 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-var-lib-kubelet\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344371 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-modprobe-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344382 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-var-lib-cni-multus\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344383 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-host-run-multus-certs\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344399 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-etc-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344469 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344473 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-etc-openvswitch\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.344489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344494 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cnibin\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344516 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-ovn-kubernetes\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344518 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15140033-be3b-4729-9a78-3a38b3d1ef48-host-slash\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344525 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344480 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-modprobe-d\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344481 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab-hosts-file\") pod \"node-resolver-p74tq\" (UID: \"1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab\") " pod="openshift-dns/node-resolver-p74tq" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344546 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15140033-be3b-4729-9a78-3a38b3d1ef48-host-slash\") pod \"iptables-alerter-mwhv4\" (UID: \"15140033-be3b-4729-9a78-3a38b3d1ef48\") " pod="openshift-network-operator/iptables-alerter-mwhv4" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344552 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-registration-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344582 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-cnibin\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344612 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344653 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-registration-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344657 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344699 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-kubelet-config\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344738 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b654842-b7a7-4e8d-8c92-8729111d0373-kubelet-config\") pod \"global-pull-secret-syncer-5sqtd\" (UID: \"6b654842-b7a7-4e8d-8c92-8729111d0373\") " pod="kube-system/global-pull-secret-syncer-5sqtd" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344762 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-sys\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344785 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faee8bf1-c317-4506-a5a7-e7317ab3909d-sys\") pod \"node-exporter-ftfsr\" (UID: \"faee8bf1-c317-4506-a5a7-e7317ab3909d\") " pod="openshift-monitoring/node-exporter-ftfsr" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344784 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-conf-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344807 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-conf-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344818 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-host\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344840 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344872 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-log-socket\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344896 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-systemd\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344908 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-host\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344920 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-systemd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344938 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-systemd\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344942 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344946 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-system-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344951 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-log-socket\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344972 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-run-systemd\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344983 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.344990 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-system-cni-dir\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345019 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-netns\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345029 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/53f48b9b-ec52-44f6-a537-e6e585762c7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6p9wv\" (UID: \"53f48b9b-ec52-44f6-a537-e6e585762c7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6p9wv" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345044 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-sys\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345067 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79323c8f-776a-4a3e-942c-a467e8672c53-host-run-netns\") pod \"ovnkube-node-p682z\" (UID: \"79323c8f-776a-4a3e-942c-a467e8672c53\") " pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345069 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345100 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-conf\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345112 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef62b57-79b3-4002-ba56-d41d3b74ded9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xbzbq\" (UID: \"9ef62b57-79b3-4002-ba56-d41d3b74ded9\") " pod="openshift-multus/multus-additional-cni-plugins-xbzbq" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345134 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-sys\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345136 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c346bfd4-9367-4a93-a44e-022b444c9f8c-crio-socket\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345166 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c346bfd4-9367-4a93-a44e-022b444c9f8c-crio-socket\") pod \"insights-runtime-extractor-54p6s\" (UID: \"c346bfd4-9367-4a93-a44e-022b444c9f8c\") " pod="openshift-insights/insights-runtime-extractor-54p6s" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345184 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-socket-dir-parent\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345213 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3489239b-c17e-41ac-84f3-392b94d019e5-etc-sysctl-conf\") pod \"tuned-ddx75\" (UID: \"3489239b-c17e-41ac-84f3-392b94d019e5\") " pod="openshift-cluster-node-tuning-operator/tuned-ddx75" Oct 30 19:17:11.345750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.345257 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862b83b6-5666-4f55-a276-4cb674c0b4a8-multus-socket-dir-parent\") pod \"multus-w2lmc\" (UID: \"862b83b6-5666-4f55-a276-4cb674c0b4a8\") " pod="openshift-multus/multus-w2lmc" Oct 30 19:17:11.585721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:11.585702 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fjmnw" Oct 30 19:17:13.180850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:13.180822 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:13.298098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:13.298078 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:14.036530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:14.036499 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:16.962106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:16.962067 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:17:17.567342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:17.567321 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" Oct 30 19:17:18.118049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:18.118020 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:18.249216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:18.249193 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:18.249381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:18.249368 6514 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:17:20.725019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:20.724985 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:20.879039 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:20.878991 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:17:24.581704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:24.581610 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:24.694472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:24.694450 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:24.819872 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:24.819843 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:25.646052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:25.646019 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p682z" Oct 30 19:17:26.434460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:26.434436 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9ssdd" Oct 30 19:17:29.178762 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:29.178734 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8988ff5d4-t8x89" Oct 30 19:17:30.138192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:30.138158 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:30.246540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:30.246515 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:32.746627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.746589 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-588db67665-hvtxk"] Oct 30 19:17:32.751463 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.751445 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.762081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.762058 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588db67665-hvtxk"] Oct 30 19:17:32.840442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.840417 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:32.877671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877651 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-console-config\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877681 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-service-ca\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877699 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9444c2ce-87b9-4697-935c-9e783689b2a4-console-oauth-config\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877812 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-trusted-ca-bundle\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877890 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2h5\" (UniqueName: \"kubernetes.io/projected/9444c2ce-87b9-4697-935c-9e783689b2a4-kube-api-access-fn2h5\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877936 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9444c2ce-87b9-4697-935c-9e783689b2a4-tmp\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877954 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9444c2ce-87b9-4697-935c-9e783689b2a4-console-serving-cert\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.877985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.877969 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-oauth-serving-cert\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.944939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.944919 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:32.978986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.978958 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-service-ca\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.978986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.978991 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9444c2ce-87b9-4697-935c-9e783689b2a4-console-oauth-config\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979019 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-trusted-ca-bundle\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979042 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2h5\" (UniqueName: \"kubernetes.io/projected/9444c2ce-87b9-4697-935c-9e783689b2a4-kube-api-access-fn2h5\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979074 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9444c2ce-87b9-4697-935c-9e783689b2a4-tmp\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979090 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9444c2ce-87b9-4697-935c-9e783689b2a4-console-serving-cert\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979103 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-oauth-serving-cert\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979136 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-console-config\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979426 6514 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:17:32.979679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979653 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9444c2ce-87b9-4697-935c-9e783689b2a4-tmp\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979829 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-console-config\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979829 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-service-ca\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.979964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.979930 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-oauth-serving-cert\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.980222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.980200 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9444c2ce-87b9-4697-935c-9e783689b2a4-trusted-ca-bundle\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.981558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.981541 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9444c2ce-87b9-4697-935c-9e783689b2a4-console-oauth-config\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.981800 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.981780 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9444c2ce-87b9-4697-935c-9e783689b2a4-console-serving-cert\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:32.987460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:32.987434 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2h5\" (UniqueName: \"kubernetes.io/projected/9444c2ce-87b9-4697-935c-9e783689b2a4-kube-api-access-fn2h5\") pod \"console-588db67665-hvtxk\" (UID: \"9444c2ce-87b9-4697-935c-9e783689b2a4\") " pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:33.061267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:33.061221 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:33.334273 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:33.334213 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588db67665-hvtxk"] Oct 30 19:17:33.337681 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:33.337656 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9444c2ce_87b9_4697_935c_9e783689b2a4.slice/crio-4bb12974e29fc0673f6445b58f6ec096f5c31f68731da66e472f903a50ef6256 WatchSource:0}: Error finding container 4bb12974e29fc0673f6445b58f6ec096f5c31f68731da66e472f903a50ef6256: Status 404 returned error can't find the container with id 4bb12974e29fc0673f6445b58f6ec096f5c31f68731da66e472f903a50ef6256 Oct 30 19:17:34.295398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.295361 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588db67665-hvtxk" event={"ID":"9444c2ce-87b9-4697-935c-9e783689b2a4","Type":"ContainerStarted","Data":"2010ade408a92c4d3d50076e9e8ca1928e28b0d87d5cdf451becff386e9afd36"} Oct 30 19:17:34.295755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.295416 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588db67665-hvtxk" event={"ID":"9444c2ce-87b9-4697-935c-9e783689b2a4","Type":"ContainerStarted","Data":"4bb12974e29fc0673f6445b58f6ec096f5c31f68731da66e472f903a50ef6256"} Oct 30 19:17:34.422111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422063 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-588db67665-hvtxk" podStartSLOduration=2.422049565 podStartE2EDuration="2.422049565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:34.327948204 +0000 UTC m=+24.285767389" watchObservedRunningTime="2026-10-30 19:17:34.422049565 +0000 UTC m=+24.379868752" Oct 30 19:17:34.422308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422289 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:17:34.422593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422566 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="alertmanager" containerID="cri-o://039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98" gracePeriod=120 Oct 30 19:17:34.422692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422611 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-metric" containerID="cri-o://1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2" gracePeriod=120 Oct 30 19:17:34.422768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422617 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy" containerID="cri-o://fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e" gracePeriod=120 Oct 30 19:17:34.422768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422733 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="config-reloader" containerID="cri-o://7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e" gracePeriod=120 Oct 30 19:17:34.422887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422812 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="prom-label-proxy" containerID="cri-o://bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df" gracePeriod=120 Oct 30 19:17:34.423776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.422996 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-web" containerID="cri-o://d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784" gracePeriod=120 Oct 30 19:17:34.521460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.521433 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:34.634533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:34.634488 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:35.302289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302257 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df" exitCode=0 Oct 30 19:17:35.302289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302284 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e" exitCode=0 Oct 30 19:17:35.302289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302293 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e" exitCode=0 Oct 30 19:17:35.302812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302302 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98" exitCode=0 Oct 30 19:17:35.302812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302329 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df"} Oct 30 19:17:35.302812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302391 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e"} Oct 30 19:17:35.302812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302422 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e"} Oct 30 19:17:35.302812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.302467 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98"} Oct 30 19:17:35.750359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.750335 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:35.796572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796548 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796585 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796633 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796668 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796701 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796747 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796787 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.796846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796827 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.797053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796874 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.797053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796913 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.797053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.796945 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.797194 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.797150 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:35.797509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.797478 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:35.797617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.797540 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.797617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.797581 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") pod \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\" (UID: \"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5\") " Oct 30 19:17:35.797851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.797833 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-metrics-client-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.797955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.797856 6514 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.800022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.799981 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:35.800463 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.800153 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:35.801234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.801191 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.802019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.801984 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "kube-api-access-sc85n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:35.802116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.802037 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:35.802285 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.802257 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.802407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.802361 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.803217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.803183 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.803413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.803388 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.806443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.806420 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.814202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.814177 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config" pod "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" (UID: "d73dfeab-ee8f-47c9-b1b6-1209a37deaa5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:35.899234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899173 6514 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899207 6514 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-main-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899217 6514 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899227 6514 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-cluster-tls-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899236 6514 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-config-out\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899245 6514 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899255 6514 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-web-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899263 6514 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-tls-assets\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899272 6514 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899280 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc85n\" (UniqueName: \"kubernetes.io/projected/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-kube-api-access-sc85n\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:35.899446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:35.899288 6514 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5-alertmanager-main-db\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.156818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.156751 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56c4467959-2ccth" podUID="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" containerName="console" containerID="cri-o://d39744170965de47e2c3f6a1556277dccbc9ba03e34a7524db4615400393a169" gracePeriod=15 Oct 30 19:17:36.307305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.307274 6514 generic.go:356] "Generic (PLEG): container finished" podID="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" containerID="d39744170965de47e2c3f6a1556277dccbc9ba03e34a7524db4615400393a169" exitCode=0 Oct 30 19:17:36.307672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.307326 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c4467959-2ccth" event={"ID":"45afd0c1-8540-4a90-a7ee-1db0805c9e2c","Type":"ContainerDied","Data":"d39744170965de47e2c3f6a1556277dccbc9ba03e34a7524db4615400393a169"} Oct 30 19:17:36.310785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310758 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2" exitCode=0 Oct 30 19:17:36.310785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310782 6514 generic.go:356] "Generic (PLEG): container finished" podID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerID="d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784" exitCode=0 Oct 30 19:17:36.310984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310830 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2"} Oct 30 19:17:36.310984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310847 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.310984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310911 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784"} Oct 30 19:17:36.310984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310941 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d73dfeab-ee8f-47c9-b1b6-1209a37deaa5","Type":"ContainerDied","Data":"03fa645e040b2f538d1662a93bbe6f3c0b7c413d4e80575cf0528d21e23a7715"} Oct 30 19:17:36.310984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.310970 6514 scope.go:122] "RemoveContainer" containerID="bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df" Oct 30 19:17:36.320629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.320611 6514 scope.go:122] "RemoveContainer" containerID="1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2" Oct 30 19:17:36.329192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.329177 6514 scope.go:122] "RemoveContainer" containerID="fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e" Oct 30 19:17:36.337146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.337127 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:17:36.341646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.341625 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:17:36.341971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.341955 6514 scope.go:122] "RemoveContainer" containerID="d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784" Oct 30 19:17:36.349235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.349221 6514 scope.go:122] "RemoveContainer" containerID="7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e" Oct 30 19:17:36.356702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.356682 6514 scope.go:122] "RemoveContainer" containerID="039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98" Oct 30 19:17:36.363676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.363656 6514 scope.go:122] "RemoveContainer" containerID="105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee" Oct 30 19:17:36.368773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.368754 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:17:36.369090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369074 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369092 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369109 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="alertmanager" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369116 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="alertmanager" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369126 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="config-reloader" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369133 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="config-reloader" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369146 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="init-config-reloader" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369153 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="init-config-reloader" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369193 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-web" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369200 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-web" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369216 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-metric" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369223 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-metric" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369914 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="prom-label-proxy" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.369926 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="prom-label-proxy" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370005 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="prom-label-proxy" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370017 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370026 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="alertmanager" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370035 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="config-reloader" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370042 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="init-config-reloader" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370051 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-web" Oct 30 19:17:36.370326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370063 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" containerName="kube-rbac-proxy-metric" Oct 30 19:17:36.371474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.370836 6514 scope.go:122] "RemoveContainer" containerID="bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df" Oct 30 19:17:36.371888 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.371846 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df\": container with ID starting with bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df not found: ID does not exist" containerID="bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df" Oct 30 19:17:36.371958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.371898 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df"} err="failed to get container status \"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df\": rpc error: code = NotFound desc = could not find container \"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df\": container with ID starting with bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df not found: ID does not exist" Oct 30 19:17:36.371958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.371944 6514 scope.go:122] "RemoveContainer" containerID="1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2" Oct 30 19:17:36.372240 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.372223 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2\": container with ID starting with 1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2 not found: ID does not exist" containerID="1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2" Oct 30 19:17:36.372305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.372247 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2"} err="failed to get container status \"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2\": rpc error: code = NotFound desc = could not find container \"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2\": container with ID starting with 1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2 not found: ID does not exist" Oct 30 19:17:36.372305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.372271 6514 scope.go:122] "RemoveContainer" containerID="fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e" Oct 30 19:17:36.372529 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.372509 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e\": container with ID starting with fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e not found: ID does not exist" containerID="fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e" Oct 30 19:17:36.372576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.372538 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e"} err="failed to get container status \"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e\": rpc error: code = NotFound desc = could not find container \"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e\": container with ID starting with fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e not found: ID does not exist" Oct 30 19:17:36.372576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.372559 6514 scope.go:122] "RemoveContainer" containerID="d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784" Oct 30 19:17:36.372811 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.372792 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784\": container with ID starting with d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784 not found: ID does not exist" containerID="d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784" Oct 30 19:17:36.372858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.372817 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784"} err="failed to get container status \"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784\": rpc error: code = NotFound desc = could not find container \"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784\": container with ID starting with d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784 not found: ID does not exist" Oct 30 19:17:36.372858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.372835 6514 scope.go:122] "RemoveContainer" containerID="7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e" Oct 30 19:17:36.373076 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.373057 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e\": container with ID starting with 7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e not found: ID does not exist" containerID="7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e" Oct 30 19:17:36.373125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373080 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e"} err="failed to get container status \"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e\": rpc error: code = NotFound desc = could not find container \"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e\": container with ID starting with 7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e not found: ID does not exist" Oct 30 19:17:36.373125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373093 6514 scope.go:122] "RemoveContainer" containerID="039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98" Oct 30 19:17:36.373311 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.373291 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98\": container with ID starting with 039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98 not found: ID does not exist" containerID="039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98" Oct 30 19:17:36.373353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373318 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98"} err="failed to get container status \"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98\": rpc error: code = NotFound desc = could not find container \"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98\": container with ID starting with 039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98 not found: ID does not exist" Oct 30 19:17:36.373353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373337 6514 scope.go:122] "RemoveContainer" containerID="105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee" Oct 30 19:17:36.373557 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:36.373543 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee\": container with ID starting with 105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee not found: ID does not exist" containerID="105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee" Oct 30 19:17:36.373600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373560 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee"} err="failed to get container status \"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee\": rpc error: code = NotFound desc = could not find container \"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee\": container with ID starting with 105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee not found: ID does not exist" Oct 30 19:17:36.373600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373571 6514 scope.go:122] "RemoveContainer" containerID="bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df" Oct 30 19:17:36.373777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373752 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df"} err="failed to get container status \"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df\": rpc error: code = NotFound desc = could not find container \"bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df\": container with ID starting with bd6900046ae54b9bb1bf26a39166fa8f23298dd85560bfe2bcd67253107a17df not found: ID does not exist" Oct 30 19:17:36.373777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.373775 6514 scope.go:122] "RemoveContainer" containerID="1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2" Oct 30 19:17:36.374365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.374339 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2"} err="failed to get container status \"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2\": rpc error: code = NotFound desc = could not find container \"1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2\": container with ID starting with 1feadc55631441fc615b45c1044f712ff66091c69e52f87c4b396bb16004b5d2 not found: ID does not exist" Oct 30 19:17:36.374365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.374365 6514 scope.go:122] "RemoveContainer" containerID="fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e" Oct 30 19:17:36.374572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.374499 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.374707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.374679 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e"} err="failed to get container status \"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e\": rpc error: code = NotFound desc = could not find container \"fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e\": container with ID starting with fd763e80062e8f64190517dec628c0de3aac2f6e61ac6ba53d6c42710b80e09e not found: ID does not exist" Oct 30 19:17:36.374769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.374709 6514 scope.go:122] "RemoveContainer" containerID="d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784" Oct 30 19:17:36.375169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375146 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784"} err="failed to get container status \"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784\": rpc error: code = NotFound desc = could not find container \"d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784\": container with ID starting with d87c84650051084d69aa7627f9f40148730fe241d9d1f14fbcc5086dc194c784 not found: ID does not exist" Oct 30 19:17:36.375297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375170 6514 scope.go:122] "RemoveContainer" containerID="7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e" Oct 30 19:17:36.375440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375422 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e"} err="failed to get container status \"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e\": rpc error: code = NotFound desc = could not find container \"7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e\": container with ID starting with 7c5f9391cc0764ac0a69471c38e9797612718addffa38da39c898874f380fa7e not found: ID does not exist" Oct 30 19:17:36.375440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375440 6514 scope.go:122] "RemoveContainer" containerID="039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98" Oct 30 19:17:36.375686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375667 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98"} err="failed to get container status \"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98\": rpc error: code = NotFound desc = could not find container \"039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98\": container with ID starting with 039e87a0952d9dbd7c3685d8f080eb3a7bc3647d8c077fb63725ed4747c82a98 not found: ID does not exist" Oct 30 19:17:36.375686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375686 6514 scope.go:122] "RemoveContainer" containerID="105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee" Oct 30 19:17:36.375952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.375933 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee"} err="failed to get container status \"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee\": rpc error: code = NotFound desc = could not find container \"105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee\": container with ID starting with 105ed875ec4520221409f8c48f4856ba818e9f9d7afcf9932a0388c836a83aee not found: ID does not exist" Oct 30 19:17:36.379340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379304 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379319 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379319 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379356 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379363 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379330 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379363 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379399 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:17:36.379439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.379430 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hlrws\"" Oct 30 19:17:36.383757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.383739 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:17:36.387421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.387402 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:36.390132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.390107 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:17:36.504878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.504844 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.504985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.504899 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.504985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.504928 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.505075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.504999 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.505075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505040 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.505161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505084 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:36.505161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505135 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.505257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505165 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") pod \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\" (UID: \"45afd0c1-8540-4a90-a7ee-1db0805c9e2c\") " Oct 30 19:17:36.505313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505289 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505317 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b621bd4-d341-4a35-a0f5-73dcb25faae3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505352 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505408 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505329 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:36.505557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505449 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:36.505557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505505 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:36.505557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505523 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b621bd4-d341-4a35-a0f5-73dcb25faae3-config-out\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505561 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505639 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b621bd4-d341-4a35-a0f5-73dcb25faae3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505660 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn69p\" (UniqueName: \"kubernetes.io/projected/6b621bd4-d341-4a35-a0f5-73dcb25faae3-kube-api-access-tn69p\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505753 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b621bd4-d341-4a35-a0f5-73dcb25faae3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505808 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505835 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-web-config\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.505972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505957 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.506033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.505987 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b621bd4-d341-4a35-a0f5-73dcb25faae3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.506104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.506088 6514 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.506138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.506115 6514 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.506169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.506132 6514 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-tmp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.506169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.506151 6514 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-oauth-serving-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.507501 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.507474 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:36.507606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.507591 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:36.507646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.507626 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td" pod "45afd0c1-8540-4a90-a7ee-1db0805c9e2c" (UID: "45afd0c1-8540-4a90-a7ee-1db0805c9e2c"). InnerVolumeSpecName "kube-api-access-hn2td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:36.607243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607220 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.607329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607249 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b621bd4-d341-4a35-a0f5-73dcb25faae3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.607329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607264 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn69p\" (UniqueName: \"kubernetes.io/projected/6b621bd4-d341-4a35-a0f5-73dcb25faae3-kube-api-access-tn69p\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.607329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607305 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b621bd4-d341-4a35-a0f5-73dcb25faae3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607533 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607781 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-web-config\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607843 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607897 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b621bd4-d341-4a35-a0f5-73dcb25faae3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607941 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.607965 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b621bd4-d341-4a35-a0f5-73dcb25faae3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608002 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608031 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608067 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b621bd4-d341-4a35-a0f5-73dcb25faae3-config-out\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608134 6514 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-serving-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608152 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn2td\" (UniqueName: \"kubernetes.io/projected/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-kube-api-access-hn2td\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.608453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608173 6514 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45afd0c1-8540-4a90-a7ee-1db0805c9e2c-console-oauth-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:36.609107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.608468 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b621bd4-d341-4a35-a0f5-73dcb25faae3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.610223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.610189 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.610483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.610342 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b621bd4-d341-4a35-a0f5-73dcb25faae3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.610784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.610618 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b621bd4-d341-4a35-a0f5-73dcb25faae3-config-out\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.610784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.610631 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.610784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.610778 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b621bd4-d341-4a35-a0f5-73dcb25faae3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.611650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.611624 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.611716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.611638 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b621bd4-d341-4a35-a0f5-73dcb25faae3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.612097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.612070 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-web-config\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.613322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.613298 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.614622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.613844 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.616349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.616326 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b621bd4-d341-4a35-a0f5-73dcb25faae3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.617308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.617291 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn69p\" (UniqueName: \"kubernetes.io/projected/6b621bd4-d341-4a35-a0f5-73dcb25faae3-kube-api-access-tn69p\") pod \"alertmanager-main-0\" (UID: \"6b621bd4-d341-4a35-a0f5-73dcb25faae3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.693755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.693735 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:17:36.967799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:36.967773 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:17:36.969948 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:17:36.969923 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b621bd4_d341_4a35_a0f5_73dcb25faae3.slice/crio-e21f7bf24b99aac2ed453bd76ab49683ff0f41949d2fb4976c7019dc96cb3dd8 WatchSource:0}: Error finding container e21f7bf24b99aac2ed453bd76ab49683ff0f41949d2fb4976c7019dc96cb3dd8: Status 404 returned error can't find the container with id e21f7bf24b99aac2ed453bd76ab49683ff0f41949d2fb4976c7019dc96cb3dd8 Oct 30 19:17:37.316824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.316763 6514 generic.go:356] "Generic (PLEG): container finished" podID="6b621bd4-d341-4a35-a0f5-73dcb25faae3" containerID="5d8a687df295d999c74aa87a3cfa3093687ee96e0950d1bdf8c13b520e254bec" exitCode=0 Oct 30 19:17:37.317242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.316835 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerDied","Data":"5d8a687df295d999c74aa87a3cfa3093687ee96e0950d1bdf8c13b520e254bec"} Oct 30 19:17:37.317242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.316929 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"e21f7bf24b99aac2ed453bd76ab49683ff0f41949d2fb4976c7019dc96cb3dd8"} Oct 30 19:17:37.318151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.318129 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c4467959-2ccth" event={"ID":"45afd0c1-8540-4a90-a7ee-1db0805c9e2c","Type":"ContainerDied","Data":"9c34cdac2db982e4e2433110e326ae93ab69a82431bcb754346d4b227f31fc9e"} Oct 30 19:17:37.318254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.318153 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c4467959-2ccth" Oct 30 19:17:37.318254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.318168 6514 scope.go:122] "RemoveContainer" containerID="d39744170965de47e2c3f6a1556277dccbc9ba03e34a7524db4615400393a169" Oct 30 19:17:37.358260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.358142 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56c4467959-2ccth"] Oct 30 19:17:37.364257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:37.364236 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56c4467959-2ccth"] Oct 30 19:17:38.156615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.156587 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" path="/var/lib/kubelet/pods/45afd0c1-8540-4a90-a7ee-1db0805c9e2c/volumes" Oct 30 19:17:38.156984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.156971 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73dfeab-ee8f-47c9-b1b6-1209a37deaa5" path="/var/lib/kubelet/pods/d73dfeab-ee8f-47c9-b1b6-1209a37deaa5/volumes" Oct 30 19:17:38.324150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.324114 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"8134da4c4ae0a56f24bcbf3b0d5a9314c3cec4ad48fd1776a126e324fe5b2a2c"} Oct 30 19:17:38.324549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.324194 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"b293dc23a79efd59336636b81e5b330862718dab8be61388514e5ba0b4bf006a"} Oct 30 19:17:38.324549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.324236 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"3ae3a26ab0811f8d16409b639804cadd4629d8338a25557e05d437ca638fa78f"} Oct 30 19:17:38.324549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.324280 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"4d8c5593eecefc7462c598d0cb8cdde41901b66b78a08915461d7f05972be0a2"} Oct 30 19:17:38.324549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.324327 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"bf2765d72b0fde552944cdfb7fa917deaf461bd02e37877762589c3366ea6ee8"} Oct 30 19:17:38.324549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.324361 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b621bd4-d341-4a35-a0f5-73dcb25faae3","Type":"ContainerStarted","Data":"e0839ef9f7ce3ba7d9f5e589372aca3ccacec4db0c42745b44f1cec452b9b05a"} Oct 30 19:17:38.356328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:38.356272 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.340821878 podStartE2EDuration="2.356260233s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.438355ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:38.354385227 +0000 UTC m=+28.312204415" watchObservedRunningTime="2026-10-30 19:17:38.356260233 +0000 UTC m=+28.314079421" Oct 30 19:17:41.752743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:41.752712 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:41.863670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:41.863642 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:43.061879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:43.061837 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:43.062247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:43.061906 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:43.066286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:43.066267 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:43.349283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:43.349211 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588db67665-hvtxk" Oct 30 19:17:43.399220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:43.399185 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5d44cd94-pfbqm"] Oct 30 19:17:43.980102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:43.980069 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:44.039323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:44.039304 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tfj5g" Oct 30 19:17:46.308665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:46.308634 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:46.433039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:46.433008 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:46.541059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:17:46.541036 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:50.878719 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:17:50.878683 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:18:04.824756 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:04.824724 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:04.936218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:04.936187 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:08.422281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.422224 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d5d44cd94-pfbqm" podUID="b3d0014b-03d6-4271-8b1b-3749435c511f" containerName="console" containerID="cri-o://a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4" gracePeriod=15 Oct 30 19:18:08.656533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.656512 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:18:08.828952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.828927 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.828967 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829012 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829039 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829060 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829090 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829174 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829224 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") pod \"b3d0014b-03d6-4271-8b1b-3749435c511f\" (UID: \"b3d0014b-03d6-4271-8b1b-3749435c511f\") " Oct 30 19:18:08.829425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829280 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:08.829425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829297 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:08.829529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829473 6514 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.829529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829492 6514 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d0014b-03d6-4271-8b1b-3749435c511f-tmp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.829529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829474 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:08.829655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829507 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:08.829655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.829645 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:08.831220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.831191 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:08.831658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.831641 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:08.831721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.831694 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf" pod "b3d0014b-03d6-4271-8b1b-3749435c511f" (UID: "b3d0014b-03d6-4271-8b1b-3749435c511f"). InnerVolumeSpecName "kube-api-access-7c4cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:08.930447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.930418 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4cf\" (UniqueName: \"kubernetes.io/projected/b3d0014b-03d6-4271-8b1b-3749435c511f-kube-api-access-7c4cf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.930447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.930446 6514 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-trusted-ca-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.930588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.930462 6514 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-serving-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.930588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.930475 6514 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-oauth-serving-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.930588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.930490 6514 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3d0014b-03d6-4271-8b1b-3749435c511f-console-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:08.930588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:08.930505 6514 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3d0014b-03d6-4271-8b1b-3749435c511f-console-oauth-config\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:09.433948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.433918 6514 generic.go:356] "Generic (PLEG): container finished" podID="b3d0014b-03d6-4271-8b1b-3749435c511f" containerID="a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4" exitCode=0 Oct 30 19:18:09.434319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.433961 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d44cd94-pfbqm" Oct 30 19:18:09.434319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.434007 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d44cd94-pfbqm" event={"ID":"b3d0014b-03d6-4271-8b1b-3749435c511f","Type":"ContainerDied","Data":"a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4"} Oct 30 19:18:09.434319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.434058 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d44cd94-pfbqm" event={"ID":"b3d0014b-03d6-4271-8b1b-3749435c511f","Type":"ContainerDied","Data":"4beecbfda0cc805f751b02bf0e0032509056266dba69d482023d3eb36c2f70a1"} Oct 30 19:18:09.434319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.434085 6514 scope.go:122] "RemoveContainer" containerID="a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4" Oct 30 19:18:09.442822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.442806 6514 scope.go:122] "RemoveContainer" containerID="a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4" Oct 30 19:18:09.443103 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:18:09.443078 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4\": container with ID starting with a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4 not found: ID does not exist" containerID="a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4" Oct 30 19:18:09.443197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.443110 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4"} err="failed to get container status \"a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4\": rpc error: code = NotFound desc = could not find container \"a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4\": container with ID starting with a3c3e20e1ee909e3de60f72093c13fc766468cfd9fcf9fb78a7f64e954d4f2b4 not found: ID does not exist" Oct 30 19:18:09.455776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.455753 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5d44cd94-pfbqm"] Oct 30 19:18:09.459184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.459164 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5d44cd94-pfbqm"] Oct 30 19:18:09.470323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:09.470303 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:10.155718 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:10.155691 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d0014b-03d6-4271-8b1b-3749435c511f" path="/var/lib/kubelet/pods/b3d0014b-03d6-4271-8b1b-3749435c511f/volumes" Oct 30 19:18:20.878072 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:18:20.878033 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:18:25.415273 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:18:25.415241 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:50.878161 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:18:50.878127 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:19:09.511665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:09.511596 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:20.878045 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:19:20.878010 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:19:50.877739 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:19:50.877696 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:19:51.103621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:51.103593 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:51.277854 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:51.277828 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:51.437247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:51.437215 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:52.697832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.697800 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l"] Oct 30 19:19:52.698207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.698122 6514 cpu_manager.go:414] "containerMap: removing container" podUID="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" containerName="console" Oct 30 19:19:52.698207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.698136 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" containerName="console" Oct 30 19:19:52.698207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.698160 6514 cpu_manager.go:414] "containerMap: removing container" podUID="b3d0014b-03d6-4271-8b1b-3749435c511f" containerName="console" Oct 30 19:19:52.698207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.698166 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d0014b-03d6-4271-8b1b-3749435c511f" containerName="console" Oct 30 19:19:52.698330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.698218 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="45afd0c1-8540-4a90-a7ee-1db0805c9e2c" containerName="console" Oct 30 19:19:52.698330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.698227 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d0014b-03d6-4271-8b1b-3749435c511f" containerName="console" Oct 30 19:19:52.701041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.701024 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.711127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.711106 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-b5vtd\"" Oct 30 19:19:52.711365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.711346 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:19:52.711445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.711431 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:19:52.768181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.768145 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l"] Oct 30 19:19:52.796964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.796944 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4ql\" (UniqueName: \"kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.797082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.796980 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.797082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.797007 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.841649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.841626 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n"] Oct 30 19:19:52.844799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.844785 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.852418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.852397 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wcgjd\"" Oct 30 19:19:52.857053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.857012 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n"] Oct 30 19:19:52.898164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898135 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.898271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898181 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4ql\" (UniqueName: \"kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.898271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898259 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.898385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898291 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvw5k\" (UniqueName: \"kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.898385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898315 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.898385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898341 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.898630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898613 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.898669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.898645 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.909219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.909197 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4ql\" (UniqueName: \"kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:52.964192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.964128 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c"] Oct 30 19:19:52.967374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.967352 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:52.972658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972636 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:19:52.972852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972730 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:19:52.972852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972751 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:19:52.972852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972762 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:19:52.972852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972802 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:19:52.973008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972856 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h9bmh\"" Oct 30 19:19:52.973008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.972943 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:19:52.981732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.981711 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c"] Oct 30 19:19:52.999335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.999310 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.999422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.999369 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvw5k\" (UniqueName: \"kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.999422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.999408 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.999752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.999734 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:52.999788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:52.999741 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:53.015742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.015722 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:19:53.016671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.016653 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:53.021693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.021671 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvw5k\" (UniqueName: \"kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:53.100181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100144 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.100316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100192 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.100316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100210 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.100316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100275 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzp7h\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.100440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100389 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.100440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100417 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.100514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.100446 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.139211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.139192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:53.163086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.163066 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:19:53.201489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201456 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.201616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201520 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.201616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201561 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.201726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201655 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.201795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201737 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.201795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201770 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.201922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.201816 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzp7h\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.202399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.202376 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.204150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.204125 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.204443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.204424 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.204516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.204451 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.204516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.204481 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.211294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.211267 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.212073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.212050 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzp7h\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h\") pod \"istiod-openshift-gateway-7876bfcb74-wdm9c\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.278598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.278526 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:53.283655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.283642 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l"] Oct 30 19:19:53.287770 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:19:53.287744 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10842a4_ca68_465f_a0dc_8b1f9be6702d.slice/crio-54581c4bf0e982a24fd504490b760891e179eaacdeb4516d357ae6f4ae365bb3 WatchSource:0}: Error finding container 54581c4bf0e982a24fd504490b760891e179eaacdeb4516d357ae6f4ae365bb3: Status 404 returned error can't find the container with id 54581c4bf0e982a24fd504490b760891e179eaacdeb4516d357ae6f4ae365bb3 Oct 30 19:19:53.289551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.289537 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:19:53.435661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.435514 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n"] Oct 30 19:19:53.438238 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:19:53.438210 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e75c39_4097_49b9_9b42_26bbe10bd4e3.slice/crio-7fb5a7bae7ecfb1d29108a24db90dde443c68bb5217bf43a641e5a6de8dba9b9 WatchSource:0}: Error finding container 7fb5a7bae7ecfb1d29108a24db90dde443c68bb5217bf43a641e5a6de8dba9b9: Status 404 returned error can't find the container with id 7fb5a7bae7ecfb1d29108a24db90dde443c68bb5217bf43a641e5a6de8dba9b9 Oct 30 19:19:53.551472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.551449 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c"] Oct 30 19:19:53.553679 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:19:53.553656 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20baf657_b21f_4ab9_bf90_1a57c3b46691.slice/crio-fba8fae1a649f32e8b014f484605ce7ff131ab2c1e6dbd1d84419659de8fbe05 WatchSource:0}: Error finding container fba8fae1a649f32e8b014f484605ce7ff131ab2c1e6dbd1d84419659de8fbe05: Status 404 returned error can't find the container with id fba8fae1a649f32e8b014f484605ce7ff131ab2c1e6dbd1d84419659de8fbe05 Oct 30 19:19:53.760488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.760462 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" event={"ID":"20baf657-b21f-4ab9-bf90-1a57c3b46691","Type":"ContainerStarted","Data":"fba8fae1a649f32e8b014f484605ce7ff131ab2c1e6dbd1d84419659de8fbe05"} Oct 30 19:19:53.761548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.761524 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" event={"ID":"42e75c39-4097-49b9-9b42-26bbe10bd4e3","Type":"ContainerStarted","Data":"7fb5a7bae7ecfb1d29108a24db90dde443c68bb5217bf43a641e5a6de8dba9b9"} Oct 30 19:19:53.762518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:53.762499 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" event={"ID":"e10842a4-ca68-465f-a0dc-8b1f9be6702d","Type":"ContainerStarted","Data":"54581c4bf0e982a24fd504490b760891e179eaacdeb4516d357ae6f4ae365bb3"} Oct 30 19:19:54.455628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:54.455593 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:57.121072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.121035 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:19:57.121423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.121118 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:19:57.732438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.732405 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:57.792506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.789425 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" event={"ID":"20baf657-b21f-4ab9-bf90-1a57c3b46691","Type":"ContainerStarted","Data":"68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274"} Oct 30 19:19:57.792506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.789754 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:19:57.822574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.822529 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" podStartSLOduration=2.257888023 podStartE2EDuration="5.822517859s" totalImagesPullingTime="3.564629836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:57.820985966 +0000 UTC m=+167.778805155" watchObservedRunningTime="2026-10-30 19:19:57.822517859 +0000 UTC m=+167.780337041" Oct 30 19:19:57.885433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:57.885399 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:58.624411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:58.624375 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:58.754031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:58.753979 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:58.800101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:58.800066 6514 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-wdm9c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:19:58.800272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:58.800129 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" podUID="20baf657-b21f-4ab9-bf90-1a57c3b46691" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:19:58.888723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:19:58.888576 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:00.804343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:00.804314 6514 generic.go:356] "Generic (PLEG): container finished" podID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerID="d148d6336c45a4e7e5b463c5602bf7c38ac1d8ee5cf972d85286da62f2fef937" exitCode=0 Oct 30 19:20:00.804898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:00.804400 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" event={"ID":"42e75c39-4097-49b9-9b42-26bbe10bd4e3","Type":"ContainerDied","Data":"d148d6336c45a4e7e5b463c5602bf7c38ac1d8ee5cf972d85286da62f2fef937"} Oct 30 19:20:00.806000 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:00.805982 6514 generic.go:356] "Generic (PLEG): container finished" podID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerID="3a1c91be63a5ae24d46017dc3f9eb6d46c5acf4ce022352c51176966382df9ed" exitCode=0 Oct 30 19:20:00.806063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:00.806027 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" event={"ID":"e10842a4-ca68-465f-a0dc-8b1f9be6702d","Type":"ContainerDied","Data":"3a1c91be63a5ae24d46017dc3f9eb6d46c5acf4ce022352c51176966382df9ed"} Oct 30 19:20:01.437790 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:01.437759 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:01.796005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:01.795984 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:20:01.811352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:01.811325 6514 generic.go:356] "Generic (PLEG): container finished" podID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerID="406bdfb694db48232c7d7d4d05894480cdbe6b4682d9fc3f7497b34e927f6529" exitCode=0 Oct 30 19:20:01.811716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:01.811411 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" event={"ID":"42e75c39-4097-49b9-9b42-26bbe10bd4e3","Type":"ContainerDied","Data":"406bdfb694db48232c7d7d4d05894480cdbe6b4682d9fc3f7497b34e927f6529"} Oct 30 19:20:03.822556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:03.822473 6514 generic.go:356] "Generic (PLEG): container finished" podID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerID="6675004837e0bc00ae6a9e82a04cfb5d0ebb0f965f3d12c798fcd3cc9a859bf8" exitCode=0 Oct 30 19:20:03.822556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:03.822543 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" event={"ID":"e10842a4-ca68-465f-a0dc-8b1f9be6702d","Type":"ContainerDied","Data":"6675004837e0bc00ae6a9e82a04cfb5d0ebb0f965f3d12c798fcd3cc9a859bf8"} Oct 30 19:20:08.849238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:08.849200 6514 generic.go:356] "Generic (PLEG): container finished" podID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerID="ad61b7be89365cb35dacc8b821bdafb29baff477486a1e2202943f580046cc9d" exitCode=0 Oct 30 19:20:08.849667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:08.849286 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" event={"ID":"42e75c39-4097-49b9-9b42-26bbe10bd4e3","Type":"ContainerDied","Data":"ad61b7be89365cb35dacc8b821bdafb29baff477486a1e2202943f580046cc9d"} Oct 30 19:20:09.568298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:09.568260 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:09.981843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:09.981817 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:20:10.157236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.157211 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util\") pod \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " Oct 30 19:20:10.157369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.157284 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle\") pod \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " Oct 30 19:20:10.157369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.157314 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k\" (UniqueName: \"kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k\") pod \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\" (UID: \"42e75c39-4097-49b9-9b42-26bbe10bd4e3\") " Oct 30 19:20:10.157979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.157948 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle" pod "42e75c39-4097-49b9-9b42-26bbe10bd4e3" (UID: "42e75c39-4097-49b9-9b42-26bbe10bd4e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:10.159522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.159493 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k" pod "42e75c39-4097-49b9-9b42-26bbe10bd4e3" (UID: "42e75c39-4097-49b9-9b42-26bbe10bd4e3"). InnerVolumeSpecName "kube-api-access-kvw5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:10.162931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.162898 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util" pod "42e75c39-4097-49b9-9b42-26bbe10bd4e3" (UID: "42e75c39-4097-49b9-9b42-26bbe10bd4e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:10.258078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.258058 6514 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:10.258078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.258076 6514 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42e75c39-4097-49b9-9b42-26bbe10bd4e3-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:10.258173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.258086 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvw5k\" (UniqueName: \"kubernetes.io/projected/42e75c39-4097-49b9-9b42-26bbe10bd4e3-kube-api-access-kvw5k\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:10.858748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.858725 6514 generic.go:356] "Generic (PLEG): container finished" podID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerID="183fc2d0182af935e2a99dedf9190432e850c33318f65dbbd1e064d5f63d8282" exitCode=0 Oct 30 19:20:10.858854 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.858807 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" event={"ID":"e10842a4-ca68-465f-a0dc-8b1f9be6702d","Type":"ContainerDied","Data":"183fc2d0182af935e2a99dedf9190432e850c33318f65dbbd1e064d5f63d8282"} Oct 30 19:20:10.860497 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.860474 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" event={"ID":"42e75c39-4097-49b9-9b42-26bbe10bd4e3","Type":"ContainerDied","Data":"7fb5a7bae7ecfb1d29108a24db90dde443c68bb5217bf43a641e5a6de8dba9b9"} Oct 30 19:20:10.860578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.860500 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716jg7n" Oct 30 19:20:10.860578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:10.860511 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb5a7bae7ecfb1d29108a24db90dde443c68bb5217bf43a641e5a6de8dba9b9" Oct 30 19:20:11.978596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:11.978570 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:20:12.069828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.069799 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util\") pod \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " Oct 30 19:20:12.069942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.069857 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql\" (UniqueName: \"kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql\") pod \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " Oct 30 19:20:12.069942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.069920 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle\") pod \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\" (UID: \"e10842a4-ca68-465f-a0dc-8b1f9be6702d\") " Oct 30 19:20:12.070277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.070252 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle" pod "e10842a4-ca68-465f-a0dc-8b1f9be6702d" (UID: "e10842a4-ca68-465f-a0dc-8b1f9be6702d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:12.071835 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.071806 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql" pod "e10842a4-ca68-465f-a0dc-8b1f9be6702d" (UID: "e10842a4-ca68-465f-a0dc-8b1f9be6702d"). InnerVolumeSpecName "kube-api-access-rx4ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:12.073555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.073532 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util" pod "e10842a4-ca68-465f-a0dc-8b1f9be6702d" (UID: "e10842a4-ca68-465f-a0dc-8b1f9be6702d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:12.170565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.170545 6514 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-util\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:12.170565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.170565 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4ql\" (UniqueName: \"kubernetes.io/projected/e10842a4-ca68-465f-a0dc-8b1f9be6702d-kube-api-access-rx4ql\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:12.170674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.170576 6514 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e10842a4-ca68-465f-a0dc-8b1f9be6702d-bundle\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:12.869784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.869763 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" Oct 30 19:20:12.869908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.869782 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448xf5l" event={"ID":"e10842a4-ca68-465f-a0dc-8b1f9be6702d","Type":"ContainerDied","Data":"54581c4bf0e982a24fd504490b760891e179eaacdeb4516d357ae6f4ae365bb3"} Oct 30 19:20:12.869908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:12.869823 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54581c4bf0e982a24fd504490b760891e179eaacdeb4516d357ae6f4ae365bb3" Oct 30 19:20:20.861622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:20.861592 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:20.878834 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:20:20.878801 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:20:21.024607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.024581 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:21.408637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.408600 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:21.734300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.734269 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:21.757240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757217 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv"] Oct 30 19:20:21.757604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757588 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="extract" Oct 30 19:20:21.757650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757606 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="extract" Oct 30 19:20:21.757650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757622 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="pull" Oct 30 19:20:21.757650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757629 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="pull" Oct 30 19:20:21.757650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757643 6514 cpu_manager.go:414] "containerMap: removing container" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="extract" Oct 30 19:20:21.757650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757648 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="extract" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757658 6514 cpu_manager.go:414] "containerMap: removing container" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="pull" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757665 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="pull" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757678 6514 cpu_manager.go:414] "containerMap: removing container" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="util" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757682 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="util" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757692 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="util" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757696 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="util" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757743 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10842a4-ca68-465f-a0dc-8b1f9be6702d" containerName="extract" Oct 30 19:20:21.757801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.757755 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e75c39-4097-49b9-9b42-26bbe10bd4e3" containerName="extract" Oct 30 19:20:21.771043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.771013 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:21.839750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.839729 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv"] Oct 30 19:20:21.860997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.860973 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:20:21.862339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.862294 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:20:21.862752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.862450 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:20:21.862752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.862602 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:20:21.863274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.863249 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hbls9\"" Oct 30 19:20:21.863274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.863256 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:20:21.935281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.935257 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:21.935391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.935288 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:21.935391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.935307 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9vx\" (UniqueName: \"kubernetes.io/projected/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-kube-api-access-pl9vx\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:21.935391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.935371 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:21.935516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.935411 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:21.935516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:21.935435 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.036526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.036462 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.036526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.036499 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9vx\" (UniqueName: \"kubernetes.io/projected/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-kube-api-access-pl9vx\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.036680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.036537 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.036680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.036581 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.036680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.036617 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.036818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.036693 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.037067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.037043 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.038898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.038851 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.039004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.038925 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.039133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.039116 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.039256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.039233 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.047661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.047636 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9vx\" (UniqueName: \"kubernetes.io/projected/c9ee772c-a7b0-469f-bc85-bcc6c5d74a91-kube-api-access-pl9vx\") pod \"opendatahub-operator-controller-manager-68fb5c5984-7qpdv\" (UID: \"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.099332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.099308 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:22.106348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.106329 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:22.236217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.236181 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:22.360732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.360646 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:22.458428 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:20:22.458400 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ee772c_a7b0_469f_bc85_bcc6c5d74a91.slice/crio-44573f22ce10f9200c4ec8e5f7bdd1b10f974287c83120cdd6e64b7d00d808d5 WatchSource:0}: Error finding container 44573f22ce10f9200c4ec8e5f7bdd1b10f974287c83120cdd6e64b7d00d808d5: Status 404 returned error can't find the container with id 44573f22ce10f9200c4ec8e5f7bdd1b10f974287c83120cdd6e64b7d00d808d5 Oct 30 19:20:22.460657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.460634 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv"] Oct 30 19:20:22.906796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:22.906766 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" event={"ID":"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91","Type":"ContainerStarted","Data":"44573f22ce10f9200c4ec8e5f7bdd1b10f974287c83120cdd6e64b7d00d808d5"} Oct 30 19:20:24.616839 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:24.616804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.729452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:24.729412 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:25.921082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:25.921057 6514 generic.go:356] "Generic (PLEG): container finished" podID="c9ee772c-a7b0-469f-bc85-bcc6c5d74a91" containerID="545f5c7e508b363d18b7a9ce74fd190d7f5bcab08924cbf528b4b55146b09d63" exitCode=0 Oct 30 19:20:25.921380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:25.921132 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" event={"ID":"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91","Type":"ContainerDied","Data":"545f5c7e508b363d18b7a9ce74fd190d7f5bcab08924cbf528b4b55146b09d63"} Oct 30 19:20:26.927215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:26.927180 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" event={"ID":"c9ee772c-a7b0-469f-bc85-bcc6c5d74a91","Type":"ContainerStarted","Data":"b17dd226ec7d3ad5207af40a8c8a05a118d4f3687cdba84a4118f50127f45de7"} Oct 30 19:20:26.927610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:26.927347 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:27.023495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:27.022818 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" podStartSLOduration=2.261770354 podStartE2EDuration="6.022799204s" totalImagesPullingTime="3.229160066s" totalInitContainerRuntime="531.868784ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:27.019632133 +0000 UTC m=+196.977451319" watchObservedRunningTime="2026-10-30 19:20:27.022799204 +0000 UTC m=+196.980618393" Oct 30 19:20:28.823272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:28.823239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:28.949195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:28.949132 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:32.191314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:32.191284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:36.048788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:36.048755 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:36.181045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:36.181013 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:37.933314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:37.933284 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-7qpdv" Oct 30 19:20:38.148920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:38.148890 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:39.006989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:39.006939 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:39.564077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:39.564048 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:39.875299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:39.875236 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:41.020902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:41.020850 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:42.244780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:42.244747 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.027585 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:48.027501 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.296052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:48.295977 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.727382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:48.727353 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.888262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:48.888239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:49.705472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:49.705441 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:50.802526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:50.802496 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:50.877755 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:20:50.877720 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:20:52.400141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:52.400096 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:53.267753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:53.267666 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:55.501303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:55.501263 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:55.995067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:55.995037 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:56.617406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:56.617377 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:58.270185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:20:58.270155 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:00.647979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:00.647944 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:00.771109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:00.771077 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:00.905158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:00.905091 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:04.395752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:04.395701 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:04.694538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:04.694506 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:05.666916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.666854 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk"] Oct 30 19:21:05.669034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.669017 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.682619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.682597 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 19:21:05.682619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.682608 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wz84h\"" Oct 30 19:21:05.685378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.685352 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk"] Oct 30 19:21:05.708636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.708610 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:05.760203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.760179 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d89664e-71a7-4b99-8761-6b0958fb85d5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.760322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.760237 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgsj\" (UniqueName: \"kubernetes.io/projected/8d89664e-71a7-4b99-8761-6b0958fb85d5-kube-api-access-6tgsj\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.760386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.760359 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d89664e-71a7-4b99-8761-6b0958fb85d5-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.773589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.773570 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 19:21:05.773796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.773757 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:21:05.773905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.773893 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:21:05.842648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.842627 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:05.860666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.860637 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgsj\" (UniqueName: \"kubernetes.io/projected/8d89664e-71a7-4b99-8761-6b0958fb85d5-kube-api-access-6tgsj\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.860771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.860674 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d89664e-71a7-4b99-8761-6b0958fb85d5-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.860771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.860721 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d89664e-71a7-4b99-8761-6b0958fb85d5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.861410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.861391 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d89664e-71a7-4b99-8761-6b0958fb85d5-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.863202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.863178 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d89664e-71a7-4b99-8761-6b0958fb85d5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.869689 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.869671 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgsj\" (UniqueName: \"kubernetes.io/projected/8d89664e-71a7-4b99-8761-6b0958fb85d5-kube-api-access-6tgsj\") pod \"kuadrant-console-plugin-66c4fb575d-pmcfk\" (UID: \"8d89664e-71a7-4b99-8761-6b0958fb85d5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:05.993958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:05.993935 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" Oct 30 19:21:06.139644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.139606 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:06.288214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.288057 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk"] Oct 30 19:21:06.292771 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:21:06.292739 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d89664e_71a7_4b99_8761_6b0958fb85d5.slice/crio-097cf8f9dfa02a64357f8749bef57afdffaf6316e159d65639aa00f511e947ea WatchSource:0}: Error finding container 097cf8f9dfa02a64357f8749bef57afdffaf6316e159d65639aa00f511e947ea: Status 404 returned error can't find the container with id 097cf8f9dfa02a64357f8749bef57afdffaf6316e159d65639aa00f511e947ea Oct 30 19:21:06.417649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.417613 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:06.509428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.509398 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl"] Oct 30 19:21:06.512317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.512292 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.529048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.529028 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:21:06.530108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.530092 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:21:06.530176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.530112 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qtqt2\"" Oct 30 19:21:06.567004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.566931 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxss\" (UniqueName: \"kubernetes.io/projected/04c85d7a-0eee-4809-831b-654ab8b490f5-kube-api-access-xwxss\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.567108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.567020 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c85d7a-0eee-4809-831b-654ab8b490f5-tls-certs\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.567108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.567074 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c85d7a-0eee-4809-831b-654ab8b490f5-tmp\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.631962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.631926 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl"] Oct 30 19:21:06.668398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.668369 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c85d7a-0eee-4809-831b-654ab8b490f5-tmp\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.668843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.668473 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxss\" (UniqueName: \"kubernetes.io/projected/04c85d7a-0eee-4809-831b-654ab8b490f5-kube-api-access-xwxss\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.668843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.668531 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c85d7a-0eee-4809-831b-654ab8b490f5-tls-certs\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.671357 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.671333 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c85d7a-0eee-4809-831b-654ab8b490f5-tmp\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.671464 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.671414 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c85d7a-0eee-4809-831b-654ab8b490f5-tls-certs\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.691035 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.691006 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxss\" (UniqueName: \"kubernetes.io/projected/04c85d7a-0eee-4809-831b-654ab8b490f5-kube-api-access-xwxss\") pod \"kube-auth-proxy-7c5f4bd96d-rhxkl\" (UID: \"04c85d7a-0eee-4809-831b-654ab8b490f5\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.739744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.739723 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:06.823769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.823693 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" Oct 30 19:21:06.911777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:06.911749 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:07.042200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:07.042168 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:07.082479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:07.082412 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" event={"ID":"8d89664e-71a7-4b99-8761-6b0958fb85d5","Type":"ContainerStarted","Data":"097cf8f9dfa02a64357f8749bef57afdffaf6316e159d65639aa00f511e947ea"} Oct 30 19:21:07.115816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:07.115746 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl"] Oct 30 19:21:07.123876 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:21:07.123840 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c85d7a_0eee_4809_831b_654ab8b490f5.slice/crio-65d5b8d57b58822a9283eecc72ca93bcea551dc06f8414a7a3d46989ef4f0fd3 WatchSource:0}: Error finding container 65d5b8d57b58822a9283eecc72ca93bcea551dc06f8414a7a3d46989ef4f0fd3: Status 404 returned error can't find the container with id 65d5b8d57b58822a9283eecc72ca93bcea551dc06f8414a7a3d46989ef4f0fd3 Oct 30 19:21:07.622892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:07.622014 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:07.758173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:07.758134 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.054351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.054282 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.091343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.091294 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" event={"ID":"04c85d7a-0eee-4809-831b-654ab8b490f5","Type":"ContainerStarted","Data":"65d5b8d57b58822a9283eecc72ca93bcea551dc06f8414a7a3d46989ef4f0fd3"} Oct 30 19:21:08.298986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.297384 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:21:08.300922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.300896 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.315670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.315647 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:21:08.334915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.334883 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:21:08.386028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.386003 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:21:08.391664 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.391629 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshfz\" (UniqueName: \"kubernetes.io/projected/90045f0f-21f2-4ce4-b3de-ec9c24c163b9-kube-api-access-kshfz\") pod \"limitador-limitador-7c9b87d574-2m8b7\" (UID: \"90045f0f-21f2-4ce4-b3de-ec9c24c163b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.391797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.391688 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90045f0f-21f2-4ce4-b3de-ec9c24c163b9-config-file\") pod \"limitador-limitador-7c9b87d574-2m8b7\" (UID: \"90045f0f-21f2-4ce4-b3de-ec9c24c163b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.439118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.439079 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.493238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.493203 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshfz\" (UniqueName: \"kubernetes.io/projected/90045f0f-21f2-4ce4-b3de-ec9c24c163b9-kube-api-access-kshfz\") pod \"limitador-limitador-7c9b87d574-2m8b7\" (UID: \"90045f0f-21f2-4ce4-b3de-ec9c24c163b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.493394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.493255 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90045f0f-21f2-4ce4-b3de-ec9c24c163b9-config-file\") pod \"limitador-limitador-7c9b87d574-2m8b7\" (UID: \"90045f0f-21f2-4ce4-b3de-ec9c24c163b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.494055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.494031 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90045f0f-21f2-4ce4-b3de-ec9c24c163b9-config-file\") pod \"limitador-limitador-7c9b87d574-2m8b7\" (UID: \"90045f0f-21f2-4ce4-b3de-ec9c24c163b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.531624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.531593 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshfz\" (UniqueName: \"kubernetes.io/projected/90045f0f-21f2-4ce4-b3de-ec9c24c163b9-kube-api-access-kshfz\") pod \"limitador-limitador-7c9b87d574-2m8b7\" (UID: \"90045f0f-21f2-4ce4-b3de-ec9c24c163b9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.603897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.603818 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.619436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.619414 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:08.801141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.801112 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.915915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:08.915820 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:21:08.925788 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:21:08.925758 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90045f0f_21f2_4ce4_b3de_ec9c24c163b9.slice/crio-17a65a3a2af8a24c7712ab5066a873f98ba3c29fb5d1e6e7d3df975425068da6 WatchSource:0}: Error finding container 17a65a3a2af8a24c7712ab5066a873f98ba3c29fb5d1e6e7d3df975425068da6: Status 404 returned error can't find the container with id 17a65a3a2af8a24c7712ab5066a873f98ba3c29fb5d1e6e7d3df975425068da6 Oct 30 19:21:09.040332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:09.040297 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:09.097859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:09.097788 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" event={"ID":"90045f0f-21f2-4ce4-b3de-ec9c24c163b9","Type":"ContainerStarted","Data":"17a65a3a2af8a24c7712ab5066a873f98ba3c29fb5d1e6e7d3df975425068da6"} Oct 30 19:21:09.646191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:09.646162 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:10.164108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:10.164086 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:21:11.114209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:11.114172 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" event={"ID":"04c85d7a-0eee-4809-831b-654ab8b490f5","Type":"ContainerStarted","Data":"7bbc85c1c8bce6f6f0c5c4dc556367d3c6e236e1c62aff73695b83fe875ac975"} Oct 30 19:21:11.137051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:11.137004 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-rhxkl" podStartSLOduration=2.10413163 podStartE2EDuration="5.136991271s" totalImagesPullingTime="3.032859641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:11.136114168 +0000 UTC m=+241.093933357" watchObservedRunningTime="2026-10-30 19:21:11.136991271 +0000 UTC m=+241.094810455" Oct 30 19:21:11.596206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:11.596176 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:11.716768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:11.716730 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:12.119788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:12.119753 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" event={"ID":"90045f0f-21f2-4ce4-b3de-ec9c24c163b9","Type":"ContainerStarted","Data":"8923e040b0bbb5693c6eb935fb656aa0864b7d32b77e228f92202e00317b6729"} Oct 30 19:21:12.140080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:12.139984 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" podStartSLOduration=1.191984773 podStartE2EDuration="4.139967244s" totalImagesPullingTime="2.947982471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.13920307 +0000 UTC m=+242.097022260" watchObservedRunningTime="2026-10-30 19:21:12.139967244 +0000 UTC m=+242.097786434" Oct 30 19:21:12.922066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:12.922011 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.037769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:13.037727 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.126021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:13.125987 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" event={"ID":"8d89664e-71a7-4b99-8761-6b0958fb85d5","Type":"ContainerStarted","Data":"7b3dcf9eb92df62ef4b5fa674b35f1b2e6d8a53e7d13d27208a32f58a69014e8"} Oct 30 19:21:13.126267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:13.126248 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:13.146707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:13.146664 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pmcfk" podStartSLOduration=2.182268838 podStartE2EDuration="8.146652997s" totalImagesPullingTime="5.964384159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:13.145332333 +0000 UTC m=+243.103151521" watchObservedRunningTime="2026-10-30 19:21:13.146652997 +0000 UTC m=+243.104472182" Oct 30 19:21:13.338907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:13.338852 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.535122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:13.535086 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.538371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:14.538342 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.661274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:14.661246 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:18.849154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:18.849123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.000564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:19.000540 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.626698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:20.626666 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.805103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:20.805074 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.878744 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:21:20.878658 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:21:23.982142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:23.982113 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:24.121849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:24.121823 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:24.131347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:24.131328 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2m8b7" Oct 30 19:21:28.002818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:28.002784 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:28.128660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:28.128635 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:28.905826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:28.905790 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.152780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:29.152741 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.671503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:29.671474 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:32.861143 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:32.861110 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:32.982779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:32.982750 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.551072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.551038 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.671459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.671430 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.797356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.797326 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.909960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.909884 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.917300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.917276 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bb5597689-7xpv4"] Oct 30 19:21:49.938446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.938421 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bb5597689-7xpv4"] Oct 30 19:21:49.938574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.938509 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:49.947105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.947077 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:21:49.947105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.947098 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5rr7g\"" Oct 30 19:21:49.947274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.947156 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:21:49.947410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.947386 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:21:49.947467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.947434 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:21:49.947467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:49.947435 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:21:50.012530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.012503 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.012722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.012548 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkh8\" (UniqueName: \"kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.012722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.012569 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.113890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.113851 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.114005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.113906 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkh8\" (UniqueName: \"kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.114005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.113922 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.116312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.116289 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.116393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.116329 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.123127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.123105 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkh8\" (UniqueName: \"kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8\") pod \"maas-controller-6bb5597689-7xpv4\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.251727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.251703 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:50.523222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:50.523159 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bb5597689-7xpv4"] Oct 30 19:21:50.526366 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:21:50.526338 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd81d55c_1bb0_4b71_8adb_0e1ccaf3efc5.slice/crio-41dd8d1520dd52ea05466d78fa6a8d26b87ebd1879609d7c2d2098ee7f7397e5 WatchSource:0}: Error finding container 41dd8d1520dd52ea05466d78fa6a8d26b87ebd1879609d7c2d2098ee7f7397e5: Status 404 returned error can't find the container with id 41dd8d1520dd52ea05466d78fa6a8d26b87ebd1879609d7c2d2098ee7f7397e5 Oct 30 19:21:50.878061 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:21:50.877984 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:21:51.275378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:51.275343 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-7xpv4" event={"ID":"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5","Type":"ContainerStarted","Data":"41dd8d1520dd52ea05466d78fa6a8d26b87ebd1879609d7c2d2098ee7f7397e5"} Oct 30 19:21:53.286301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:53.286271 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-7xpv4" event={"ID":"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5","Type":"ContainerStarted","Data":"f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6"} Oct 30 19:21:53.286672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:53.286408 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:21:53.311120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:53.311076 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6bb5597689-7xpv4" podStartSLOduration=1.621954225 podStartE2EDuration="4.311063001s" totalImagesPullingTime="2.689108776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:53.30874683 +0000 UTC m=+283.266566040" watchObservedRunningTime="2026-10-30 19:21:53.311063001 +0000 UTC m=+283.268882236" Oct 30 19:21:53.413538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:53.413506 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:53.536592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:53.536518 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:54.410469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.410439 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9"] Oct 30 19:21:54.413766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.413744 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.422488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.422466 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 19:21:54.422608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.422571 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9"] Oct 30 19:21:54.422765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.422743 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 19:21:54.437943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.437917 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 19:21:54.437943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.437937 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-57twl\"" Oct 30 19:21:54.454693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.454670 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.454817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.454701 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7gl\" (UniqueName: \"kubernetes.io/projected/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-kube-api-access-qc7gl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.454817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.454725 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.454973 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.454855 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.487917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.487890 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:54.555334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.555309 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.555448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.555339 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7gl\" (UniqueName: \"kubernetes.io/projected/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-kube-api-access-qc7gl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.555448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.555363 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.555448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.555389 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.556204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.556157 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.556390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.556370 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.557819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.557799 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.563776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.563746 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7gl\" (UniqueName: \"kubernetes.io/projected/9a6f1251-fe5c-4cbf-b306-7db1ef6200b5-kube-api-access-qc7gl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-97mp9\" (UID: \"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:54.600071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.600041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:54.728433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:54.728411 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:21:55.002266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:55.002198 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9"] Oct 30 19:21:55.005330 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:21:55.005306 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6f1251_fe5c_4cbf_b306_7db1ef6200b5.slice/crio-edfe94b8842fb91a87de983005f8db5a24d3dda68b7d980ed7fa7e5741dd5946 WatchSource:0}: Error finding container edfe94b8842fb91a87de983005f8db5a24d3dda68b7d980ed7fa7e5741dd5946: Status 404 returned error can't find the container with id edfe94b8842fb91a87de983005f8db5a24d3dda68b7d980ed7fa7e5741dd5946 Oct 30 19:21:55.295959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:21:55.295893 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" event={"ID":"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5","Type":"ContainerStarted","Data":"edfe94b8842fb91a87de983005f8db5a24d3dda68b7d980ed7fa7e5741dd5946"} Oct 30 19:22:04.282576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.282538 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.293014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.292988 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:22:04.509994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.509808 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.586073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.585993 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x2fwr"] Oct 30 19:22:04.610288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.610263 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.622769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.622673 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x2fwr"] Oct 30 19:22:04.649797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.649766 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p4x\" (UniqueName: \"kubernetes.io/projected/350a83c1-ef56-4d86-adc4-e296e3655933-kube-api-access-n8p4x\") pod \"model-serving-api-75bd89fdfc-x2fwr\" (UID: \"350a83c1-ef56-4d86-adc4-e296e3655933\") " pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.649936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.649833 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a83c1-ef56-4d86-adc4-e296e3655933-tls-certs\") pod \"model-serving-api-75bd89fdfc-x2fwr\" (UID: \"350a83c1-ef56-4d86-adc4-e296e3655933\") " pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.651893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.651858 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2bn8z\"" Oct 30 19:22:04.651893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.651885 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 19:22:04.750670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.750636 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p4x\" (UniqueName: \"kubernetes.io/projected/350a83c1-ef56-4d86-adc4-e296e3655933-kube-api-access-n8p4x\") pod \"model-serving-api-75bd89fdfc-x2fwr\" (UID: \"350a83c1-ef56-4d86-adc4-e296e3655933\") " pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.750842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.750705 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a83c1-ef56-4d86-adc4-e296e3655933-tls-certs\") pod \"model-serving-api-75bd89fdfc-x2fwr\" (UID: \"350a83c1-ef56-4d86-adc4-e296e3655933\") " pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.753297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.753269 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a83c1-ef56-4d86-adc4-e296e3655933-tls-certs\") pod \"model-serving-api-75bd89fdfc-x2fwr\" (UID: \"350a83c1-ef56-4d86-adc4-e296e3655933\") " pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.772441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.772414 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p4x\" (UniqueName: \"kubernetes.io/projected/350a83c1-ef56-4d86-adc4-e296e3655933-kube-api-access-n8p4x\") pod \"model-serving-api-75bd89fdfc-x2fwr\" (UID: \"350a83c1-ef56-4d86-adc4-e296e3655933\") " pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:04.814569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.814541 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.931418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:04.931390 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:05.008994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.008971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:05.266779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.266698 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x2fwr"] Oct 30 19:22:05.270203 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:22:05.270167 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350a83c1_ef56_4d86_adc4_e296e3655933.slice/crio-a9c469d556445f3c3f93a560df1dd4a1286c6724f16185c0c48a6e6bf7fe73be WatchSource:0}: Error finding container a9c469d556445f3c3f93a560df1dd4a1286c6724f16185c0c48a6e6bf7fe73be: Status 404 returned error can't find the container with id a9c469d556445f3c3f93a560df1dd4a1286c6724f16185c0c48a6e6bf7fe73be Oct 30 19:22:05.271196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.271109 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:05.349771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.349742 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" event={"ID":"350a83c1-ef56-4d86-adc4-e296e3655933","Type":"ContainerStarted","Data":"a9c469d556445f3c3f93a560df1dd4a1286c6724f16185c0c48a6e6bf7fe73be"} Oct 30 19:22:05.412494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.412471 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz"] Oct 30 19:22:05.417153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.417130 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.428277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.428242 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz"] Oct 30 19:22:05.456056 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456034 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.456195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456069 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.456195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456121 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68tkv\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.456294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456264 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.456347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456331 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.456395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456354 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.456395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.456381 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557100 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557142 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557180 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tkv\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557219 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557287 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557312 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557515 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.557946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.557899 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.559948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.559902 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.560244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.560219 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.560339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.560276 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.560455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.560434 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.568638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.568616 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.568769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.568752 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tkv\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mfsgz\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:05.731306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:05.731277 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:06.019787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.019749 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz"] Oct 30 19:22:06.034483 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:22:06.034450 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15316da0_3a8c_430d_a60b_dfa38e574a42.slice/crio-cbc0935825fe7ee3755c0c8ea7929240c3e966fe2baa9ab67c965be13f1f2095 WatchSource:0}: Error finding container cbc0935825fe7ee3755c0c8ea7929240c3e966fe2baa9ab67c965be13f1f2095: Status 404 returned error can't find the container with id cbc0935825fe7ee3755c0c8ea7929240c3e966fe2baa9ab67c965be13f1f2095 Oct 30 19:22:06.038535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.038109 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:22:06.038535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.038182 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:22:06.068534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.067785 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.213160 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.213119 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.361342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.361257 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" event={"ID":"15316da0-3a8c-430d-a60b-dfa38e574a42","Type":"ContainerStarted","Data":"97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511"} Oct 30 19:22:06.361342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.361321 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" event={"ID":"15316da0-3a8c-430d-a60b-dfa38e574a42","Type":"ContainerStarted","Data":"cbc0935825fe7ee3755c0c8ea7929240c3e966fe2baa9ab67c965be13f1f2095"} Oct 30 19:22:06.361906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.361379 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:06.447565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:06.447527 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:07.336259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:07.336235 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:07.396030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:07.395419 6514 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mfsgz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:07.396030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:07.395460 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" podUID="15316da0-3a8c-430d-a60b-dfa38e574a42" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:08.374358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.374287 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" event={"ID":"350a83c1-ef56-4d86-adc4-e296e3655933","Type":"ContainerStarted","Data":"4e247ae90a655ddd0286765f78128a86dc35577900b49d9cd5e3ff27a532816b"} Oct 30 19:22:08.374516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.374419 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:08.375834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.375812 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" event={"ID":"9a6f1251-fe5c-4cbf-b306-7db1ef6200b5","Type":"ContainerStarted","Data":"6d517eb5ca24920734393a1649a3a68edc7f811605d98e0884c6d29f691ebe57"} Oct 30 19:22:08.375979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.375859 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:22:08.403778 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.403732 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" podStartSLOduration=2.364019763 podStartE2EDuration="4.403721274s" totalImagesPullingTime="2.039701511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:08.401007793 +0000 UTC m=+298.358826980" watchObservedRunningTime="2026-10-30 19:22:08.403721274 +0000 UTC m=+298.361540462" Oct 30 19:22:08.404113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.403812 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" podStartSLOduration=3.403806634 podStartE2EDuration="3.403806634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:06.4074364 +0000 UTC m=+296.365255590" watchObservedRunningTime="2026-10-30 19:22:08.403806634 +0000 UTC m=+298.361625823" Oct 30 19:22:08.432927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:08.432859 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" podStartSLOduration=1.431903777 podStartE2EDuration="14.432847358s" totalImagesPullingTime="13.000943581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:08.432133409 +0000 UTC m=+298.389952598" watchObservedRunningTime="2026-10-30 19:22:08.432847358 +0000 UTC m=+298.390666546" Oct 30 19:22:09.682288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:09.682259 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.178947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.178913 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:22:10.179220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.179202 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:22:10.369879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.369834 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:22:10.427665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.427638 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c"] Oct 30 19:22:10.427832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.427809 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" podUID="20baf657-b21f-4ab9-bf90-1a57c3b46691" containerName="discovery" containerID="cri-o://68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274" gracePeriod=30 Oct 30 19:22:10.670431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.670408 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:22:10.810832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810771 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.810832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810815 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.811307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810835 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.811307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810882 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.811307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810913 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.811307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810943 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.811307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.810989 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h\") pod \"20baf657-b21f-4ab9-bf90-1a57c3b46691\" (UID: \"20baf657-b21f-4ab9-bf90-1a57c3b46691\") " Oct 30 19:22:10.811577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.811319 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:10.813290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.813265 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:10.813416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.813274 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:10.813477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.813406 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "kube-api-access-kzp7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:10.813477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.813410 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:10.813477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.813441 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:10.815182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.815164 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts" pod "20baf657-b21f-4ab9-bf90-1a57c3b46691" (UID: "20baf657-b21f-4ab9-bf90-1a57c3b46691"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:10.911841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911811 6514 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-kubeconfig\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:10.911841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911838 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:10.912008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911852 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzp7h\" (UniqueName: \"kubernetes.io/projected/20baf657-b21f-4ab9-bf90-1a57c3b46691-kube-api-access-kzp7h\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:10.912008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911888 6514 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20baf657-b21f-4ab9-bf90-1a57c3b46691-local-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:10.912008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911902 6514 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-dns-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:10.912008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911915 6514 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20baf657-b21f-4ab9-bf90-1a57c3b46691-cacerts\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:10.912008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:10.911927 6514 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20baf657-b21f-4ab9-bf90-1a57c3b46691-istio-csr-ca-configmap\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:11.390299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.390272 6514 generic.go:356] "Generic (PLEG): container finished" podID="20baf657-b21f-4ab9-bf90-1a57c3b46691" containerID="68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274" exitCode=0 Oct 30 19:22:11.390417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.390333 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" Oct 30 19:22:11.390417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.390353 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" event={"ID":"20baf657-b21f-4ab9-bf90-1a57c3b46691","Type":"ContainerDied","Data":"68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274"} Oct 30 19:22:11.390417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.390407 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c" event={"ID":"20baf657-b21f-4ab9-bf90-1a57c3b46691","Type":"ContainerDied","Data":"fba8fae1a649f32e8b014f484605ce7ff131ab2c1e6dbd1d84419659de8fbe05"} Oct 30 19:22:11.390522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.390434 6514 scope.go:122] "RemoveContainer" containerID="68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274" Oct 30 19:22:11.399355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.399339 6514 scope.go:122] "RemoveContainer" containerID="68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274" Oct 30 19:22:11.399612 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:22:11.399596 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274\": container with ID starting with 68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274 not found: ID does not exist" containerID="68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274" Oct 30 19:22:11.399673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.399624 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274"} err="failed to get container status \"68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274\": rpc error: code = NotFound desc = could not find container \"68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274\": container with ID starting with 68cc8d9f492079b630da20ffab75ddf087e762465c3a95fc947fa7771cd46274 not found: ID does not exist" Oct 30 19:22:11.413589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.413569 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c"] Oct 30 19:22:11.417411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:11.417389 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wdm9c"] Oct 30 19:22:12.156840 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:12.156805 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20baf657-b21f-4ab9-bf90-1a57c3b46691" path="/var/lib/kubelet/pods/20baf657-b21f-4ab9-bf90-1a57c3b46691/volumes" Oct 30 19:22:19.384257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:19.384224 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-x2fwr" Oct 30 19:22:19.385537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:19.385515 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-97mp9" Oct 30 19:22:20.878221 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:22:20.878186 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:22:24.788576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:24.788543 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:24.895645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:24.895613 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.049386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.049356 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.165661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.165629 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz"] Oct 30 19:22:26.166011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.165995 6514 cpu_manager.go:414] "containerMap: removing container" podUID="20baf657-b21f-4ab9-bf90-1a57c3b46691" containerName="discovery" Oct 30 19:22:26.166066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.166011 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20baf657-b21f-4ab9-bf90-1a57c3b46691" containerName="discovery" Oct 30 19:22:26.166066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.166059 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="20baf657-b21f-4ab9-bf90-1a57c3b46691" containerName="discovery" Oct 30 19:22:26.172945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.172924 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.179299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.179265 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:22:26.200022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.199988 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:22:26.200131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.200066 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2z7lq\"" Oct 30 19:22:26.200299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.200284 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:22:26.202629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.202607 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz"] Oct 30 19:22:26.278632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.278615 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:22:26.294724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.294700 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.331133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.331068 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.331217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.331148 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sshwk\" (UniqueName: \"kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.331217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.331195 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.331307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.331247 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.412268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.412249 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.432355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.432320 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sshwk\" (UniqueName: \"kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.432437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.432370 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.432437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.432390 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.432437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.432412 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.432992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.432975 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.434789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.434771 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.434948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.434917 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.444706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.444673 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sshwk\" (UniqueName: \"kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk\") pod \"maas-api-6d669c7fcd-6m6fz\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.485983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.485963 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:26.772822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:26.772797 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz"] Oct 30 19:22:26.774294 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:22:26.774271 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80719fa_84eb_43cd_aa94_029c1261871a.slice/crio-6a6c8515e1d1b4822fe73e83c9307f271cb4f1337cdc18f02162679ba540061a WatchSource:0}: Error finding container 6a6c8515e1d1b4822fe73e83c9307f271cb4f1337cdc18f02162679ba540061a: Status 404 returned error can't find the container with id 6a6c8515e1d1b4822fe73e83c9307f271cb4f1337cdc18f02162679ba540061a Oct 30 19:22:27.173656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:27.173567 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:22:27.455164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:27.455128 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" event={"ID":"c80719fa-84eb-43cd-aa94-029c1261871a","Type":"ContainerStarted","Data":"6a6c8515e1d1b4822fe73e83c9307f271cb4f1337cdc18f02162679ba540061a"} Oct 30 19:22:29.464969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:29.464935 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" event={"ID":"c80719fa-84eb-43cd-aa94-029c1261871a","Type":"ContainerStarted","Data":"591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593"} Oct 30 19:22:29.465316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:29.465024 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:29.487413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:29.487360 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" podStartSLOduration=1.084580581 podStartE2EDuration="3.487327088s" totalImagesPullingTime="2.402746507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:29.485496282 +0000 UTC m=+319.443315469" watchObservedRunningTime="2026-10-30 19:22:29.487327088 +0000 UTC m=+319.445146277" Oct 30 19:22:35.474303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:35.474279 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:36.140156 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:36.140123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:36.568642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:36.568609 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:44.778485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:44.778452 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz"] Oct 30 19:22:44.778899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:44.778640 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" podUID="c80719fa-84eb-43cd-aa94-029c1261871a" containerName="maas-api" containerID="cri-o://591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593" gracePeriod=30 Oct 30 19:22:45.021217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.021196 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:45.166278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.166213 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs\") pod \"c80719fa-84eb-43cd-aa94-029c1261871a\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " Oct 30 19:22:45.166278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.166243 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls\") pod \"c80719fa-84eb-43cd-aa94-029c1261871a\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " Oct 30 19:22:45.166444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.166289 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk\" (UniqueName: \"kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk\") pod \"c80719fa-84eb-43cd-aa94-029c1261871a\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " Oct 30 19:22:45.166444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.166357 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca\") pod \"c80719fa-84eb-43cd-aa94-029c1261871a\" (UID: \"c80719fa-84eb-43cd-aa94-029c1261871a\") " Oct 30 19:22:45.166732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.166707 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca" pod "c80719fa-84eb-43cd-aa94-029c1261871a" (UID: "c80719fa-84eb-43cd-aa94-029c1261871a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:45.168356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.168330 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs" pod "c80719fa-84eb-43cd-aa94-029c1261871a" (UID: "c80719fa-84eb-43cd-aa94-029c1261871a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:45.168460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.168359 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk" pod "c80719fa-84eb-43cd-aa94-029c1261871a" (UID: "c80719fa-84eb-43cd-aa94-029c1261871a"). InnerVolumeSpecName "kube-api-access-sshwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:45.168460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.168433 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls" pod "c80719fa-84eb-43cd-aa94-029c1261871a" (UID: "c80719fa-84eb-43cd-aa94-029c1261871a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:45.267705 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.267679 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sshwk\" (UniqueName: \"kubernetes.io/projected/c80719fa-84eb-43cd-aa94-029c1261871a-kube-api-access-sshwk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.267705 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.267702 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80719fa-84eb-43cd-aa94-029c1261871a-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.267832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.267712 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.267832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.267721 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80719fa-84eb-43cd-aa94-029c1261871a-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.530667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.530640 6514 generic.go:356] "Generic (PLEG): container finished" podID="c80719fa-84eb-43cd-aa94-029c1261871a" containerID="591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593" exitCode=0 Oct 30 19:22:45.530752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.530690 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" Oct 30 19:22:45.530752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.530731 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" event={"ID":"c80719fa-84eb-43cd-aa94-029c1261871a","Type":"ContainerDied","Data":"591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593"} Oct 30 19:22:45.530827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.530782 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz" event={"ID":"c80719fa-84eb-43cd-aa94-029c1261871a","Type":"ContainerDied","Data":"6a6c8515e1d1b4822fe73e83c9307f271cb4f1337cdc18f02162679ba540061a"} Oct 30 19:22:45.530827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.530806 6514 scope.go:122] "RemoveContainer" containerID="591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593" Oct 30 19:22:45.539504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.539486 6514 scope.go:122] "RemoveContainer" containerID="591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593" Oct 30 19:22:45.539764 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:22:45.539747 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593\": container with ID starting with 591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593 not found: ID does not exist" containerID="591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593" Oct 30 19:22:45.539804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.539772 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593"} err="failed to get container status \"591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593\": rpc error: code = NotFound desc = could not find container \"591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593\": container with ID starting with 591edc386e66cbe63684987222a24bc9a57cc8b061d3af5ba5973d6b99466593 not found: ID does not exist" Oct 30 19:22:45.553292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.553272 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz"] Oct 30 19:22:45.557626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:45.557602 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6m6fz"] Oct 30 19:22:46.156874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:22:46.156835 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80719fa-84eb-43cd-aa94-029c1261871a" path="/var/lib/kubelet/pods/c80719fa-84eb-43cd-aa94-029c1261871a/volumes" Oct 30 19:22:50.878089 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:22:50.878053 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:23:09.717986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:09.717948 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:16.683781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:16.683742 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:20.121726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:20.121693 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:20.252036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:20.252001 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:20.879478 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:23:20.879440 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:23:35.080185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:35.080097 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:35.509598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:35.509564 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:50.880114 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:23:50.880081 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:23:54.046819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.046787 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7"] Oct 30 19:23:54.047375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.047355 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c80719fa-84eb-43cd-aa94-029c1261871a" containerName="maas-api" Oct 30 19:23:54.047459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.047379 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80719fa-84eb-43cd-aa94-029c1261871a" containerName="maas-api" Oct 30 19:23:54.047516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.047482 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80719fa-84eb-43cd-aa94-029c1261871a" containerName="maas-api" Oct 30 19:23:54.049651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.049628 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.058688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.058662 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 19:23:54.058822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.058768 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:23:54.058924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.058845 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w2kzv\"" Oct 30 19:23:54.059984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.059962 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:23:54.064994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.064976 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7"] Oct 30 19:23:54.128287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.128256 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:54.148689 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.148667 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.148797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.148696 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.148797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.148717 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d55749f-62cd-4021-b995-5e9245cdbaad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.148797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.148778 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.148910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.148859 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx599\" (UniqueName: \"kubernetes.io/projected/5d55749f-62cd-4021-b995-5e9245cdbaad-kube-api-access-dx599\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.148956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.148941 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249473 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249445 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249478 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx599\" (UniqueName: \"kubernetes.io/projected/5d55749f-62cd-4021-b995-5e9245cdbaad-kube-api-access-dx599\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249502 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249578 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249603 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249626 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d55749f-62cd-4021-b995-5e9245cdbaad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.249974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249952 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.250031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.249995 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.250069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.250022 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.251910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.251889 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d55749f-62cd-4021-b995-5e9245cdbaad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.252090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.252074 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d55749f-62cd-4021-b995-5e9245cdbaad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.261071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.261047 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx599\" (UniqueName: \"kubernetes.io/projected/5d55749f-62cd-4021-b995-5e9245cdbaad-kube-api-access-dx599\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7\" (UID: \"5d55749f-62cd-4021-b995-5e9245cdbaad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.361813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.361751 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:54.669366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.669343 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7"] Oct 30 19:23:54.671343 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:23:54.671315 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d55749f_62cd_4021_b995_5e9245cdbaad.slice/crio-e03c235cf3829d7a159c77889330eef03660736f68aab894360c17a2385c243b WatchSource:0}: Error finding container e03c235cf3829d7a159c77889330eef03660736f68aab894360c17a2385c243b: Status 404 returned error can't find the container with id e03c235cf3829d7a159c77889330eef03660736f68aab894360c17a2385c243b Oct 30 19:23:54.754622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.754590 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:23:54.821789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:54.821759 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" event={"ID":"5d55749f-62cd-4021-b995-5e9245cdbaad","Type":"ContainerStarted","Data":"e03c235cf3829d7a159c77889330eef03660736f68aab894360c17a2385c243b"} Oct 30 19:23:55.388048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.388020 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb"] Oct 30 19:23:55.392575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.392539 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.404651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.404180 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 19:23:55.405723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.405671 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb"] Oct 30 19:23:55.464348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.464310 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.464541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.464362 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.464541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.464396 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgsv\" (UniqueName: \"kubernetes.io/projected/f37f7bbb-a255-4242-8332-98b97a2a0f68-kube-api-access-srgsv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.464541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.464459 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.464541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.464481 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f37f7bbb-a255-4242-8332-98b97a2a0f68-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.464759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.464565 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.565712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.565679 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.565892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.565721 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.565892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.565745 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgsv\" (UniqueName: \"kubernetes.io/projected/f37f7bbb-a255-4242-8332-98b97a2a0f68-kube-api-access-srgsv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.565892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.565797 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.565892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.565822 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f37f7bbb-a255-4242-8332-98b97a2a0f68-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.566102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.565913 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.566202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.566179 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.566539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.566505 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.566659 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.566549 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.568017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.567976 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f37f7bbb-a255-4242-8332-98b97a2a0f68-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.568658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.568631 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f37f7bbb-a255-4242-8332-98b97a2a0f68-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.585895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.585854 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgsv\" (UniqueName: \"kubernetes.io/projected/f37f7bbb-a255-4242-8332-98b97a2a0f68-kube-api-access-srgsv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb\" (UID: \"f37f7bbb-a255-4242-8332-98b97a2a0f68\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.707518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.707477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:55.713400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.713374 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:55.996902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:55.996877 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb"] Oct 30 19:23:56.002058 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:23:56.002036 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf37f7bbb_a255_4242_8332_98b97a2a0f68.slice/crio-96ccd1a0635f9611f20697cc79da203ce4dbe0ad641927a17637a03d77af98b7 WatchSource:0}: Error finding container 96ccd1a0635f9611f20697cc79da203ce4dbe0ad641927a17637a03d77af98b7: Status 404 returned error can't find the container with id 96ccd1a0635f9611f20697cc79da203ce4dbe0ad641927a17637a03d77af98b7 Oct 30 19:23:56.833621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.833585 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" event={"ID":"f37f7bbb-a255-4242-8332-98b97a2a0f68","Type":"ContainerStarted","Data":"a55a4d2320eddef97908732ee4d2750d9296cd1b0dd9845e1d84fd6c25ed7d65"} Oct 30 19:23:56.834053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.833635 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" event={"ID":"f37f7bbb-a255-4242-8332-98b97a2a0f68","Type":"ContainerStarted","Data":"96ccd1a0635f9611f20697cc79da203ce4dbe0ad641927a17637a03d77af98b7"} Oct 30 19:23:56.835102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.835075 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" event={"ID":"5d55749f-62cd-4021-b995-5e9245cdbaad","Type":"ContainerStarted","Data":"5df7a0224b00221ce98c1032aaa39ff13078fd181b4cfd909208d7d3ddf63864"} Oct 30 19:23:56.871184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.871140 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podStartSLOduration=1.5503273549999999 podStartE2EDuration="1.871129161s" totalImagesPullingTime="320.801806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:56.868884675 +0000 UTC m=+406.826703864" watchObservedRunningTime="2026-10-30 19:23:56.871129161 +0000 UTC m=+406.828948350" Oct 30 19:23:56.925869 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.925832 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" podStartSLOduration=1.688777792 podStartE2EDuration="2.925824543s" totalImagesPullingTime="1.237046751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:56.925538754 +0000 UTC m=+406.883357943" watchObservedRunningTime="2026-10-30 19:23:56.925824543 +0000 UTC m=+406.883643731" Oct 30 19:23:56.949649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.949625 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz"] Oct 30 19:23:56.951987 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.951971 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:56.971052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.971037 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 19:23:56.978626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:56.978601 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz"] Oct 30 19:23:57.081743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.081713 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4552499-23ca-48dd-9210-1e79e6ad04e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.081743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.081743 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.081926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.081885 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.081926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.081917 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.082006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.081939 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.082040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.082004 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nq2\" (UniqueName: \"kubernetes.io/projected/b4552499-23ca-48dd-9210-1e79e6ad04e6-kube-api-access-76nq2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.143851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.143783 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:57.182887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.182850 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.182894 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.182921 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.183011 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nq2\" (UniqueName: \"kubernetes.io/projected/b4552499-23ca-48dd-9210-1e79e6ad04e6-kube-api-access-76nq2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.183085 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4552499-23ca-48dd-9210-1e79e6ad04e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.183107 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.183313 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.183347 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.183477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.183455 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.185222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.185201 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4552499-23ca-48dd-9210-1e79e6ad04e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.185500 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.185480 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4552499-23ca-48dd-9210-1e79e6ad04e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.202388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.202364 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nq2\" (UniqueName: \"kubernetes.io/projected/b4552499-23ca-48dd-9210-1e79e6ad04e6-kube-api-access-76nq2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz\" (UID: \"b4552499-23ca-48dd-9210-1e79e6ad04e6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.264137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.264117 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:57.362585 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.362556 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:23:57.574484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.574454 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz"] Oct 30 19:23:57.577962 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:23:57.577940 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4552499_23ca_48dd_9210_1e79e6ad04e6.slice/crio-49d5c8b95222a46f2ed4add96454f204e74ff337f95da5126a23b3f9c9478182 WatchSource:0}: Error finding container 49d5c8b95222a46f2ed4add96454f204e74ff337f95da5126a23b3f9c9478182: Status 404 returned error can't find the container with id 49d5c8b95222a46f2ed4add96454f204e74ff337f95da5126a23b3f9c9478182 Oct 30 19:23:57.713545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.713522 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:23:57.841416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.841324 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" event={"ID":"b4552499-23ca-48dd-9210-1e79e6ad04e6","Type":"ContainerStarted","Data":"49d5c8b95222a46f2ed4add96454f204e74ff337f95da5126a23b3f9c9478182"} Oct 30 19:23:57.908496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:57.908474 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:23:58.846794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:58.846762 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" event={"ID":"b4552499-23ca-48dd-9210-1e79e6ad04e6","Type":"ContainerStarted","Data":"be05429ddc3b233da9f29a4ee433d2bca4edaf07735ca0ba06d40339d6f2d398"} Oct 30 19:23:58.876268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:58.876222 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" podStartSLOduration=2.588926072 podStartE2EDuration="2.876208541s" totalImagesPullingTime="287.282469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:58.875066665 +0000 UTC m=+408.832885849" watchObservedRunningTime="2026-10-30 19:23:58.876208541 +0000 UTC m=+408.834027729" Oct 30 19:23:59.265075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:59.265037 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:23:59.705551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:23:59.705518 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:00.380754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.380721 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd"] Oct 30 19:24:00.383413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.383393 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.388975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.388955 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 19:24:00.401811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.401786 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd"] Oct 30 19:24:00.468191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.468152 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:00.511518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.511493 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.511661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.511552 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.511729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.511701 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.511786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.511750 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c01c154-6ca2-4dcd-b558-2a3790449664-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.511847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.511820 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66f7j\" (UniqueName: \"kubernetes.io/projected/2c01c154-6ca2-4dcd-b558-2a3790449664-kube-api-access-66f7j\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.511925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.511906 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613077 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66f7j\" (UniqueName: \"kubernetes.io/projected/2c01c154-6ca2-4dcd-b558-2a3790449664-kube-api-access-66f7j\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613129 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613153 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613183 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613203 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613226 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c01c154-6ca2-4dcd-b558-2a3790449664-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613669 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613703 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.613794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.613742 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.615360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.615341 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c01c154-6ca2-4dcd-b558-2a3790449664-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.615621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.615604 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c01c154-6ca2-4dcd-b558-2a3790449664-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.622918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.622890 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66f7j\" (UniqueName: \"kubernetes.io/projected/2c01c154-6ca2-4dcd-b558-2a3790449664-kube-api-access-66f7j\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tlbtd\" (UID: \"2c01c154-6ca2-4dcd-b558-2a3790449664\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:00.696526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:00.696509 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:01.014972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:01.014931 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd"] Oct 30 19:24:01.017145 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:24:01.017118 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c01c154_6ca2_4dcd_b558_2a3790449664.slice/crio-7804be1d9d467c402ee827ea499d21f847861c7e4329c75980b6ebc955ce819b WatchSource:0}: Error finding container 7804be1d9d467c402ee827ea499d21f847861c7e4329c75980b6ebc955ce819b: Status 404 returned error can't find the container with id 7804be1d9d467c402ee827ea499d21f847861c7e4329c75980b6ebc955ce819b Oct 30 19:24:01.021639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:01.021615 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:24:01.689188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:01.689156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:01.860672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:01.860634 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" event={"ID":"2c01c154-6ca2-4dcd-b558-2a3790449664","Type":"ContainerStarted","Data":"51d44431ff3aed04daac78adf99d1f73274c03cdd1eb0c77b5176cced5b7ce47"} Oct 30 19:24:01.860803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:01.860687 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" event={"ID":"2c01c154-6ca2-4dcd-b558-2a3790449664","Type":"ContainerStarted","Data":"7804be1d9d467c402ee827ea499d21f847861c7e4329c75980b6ebc955ce819b"} Oct 30 19:24:01.882096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:01.882057 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" podStartSLOduration=1.612680338 podStartE2EDuration="1.882046121s" totalImagesPullingTime="269.365783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:01.880282133 +0000 UTC m=+411.838101322" watchObservedRunningTime="2026-10-30 19:24:01.882046121 +0000 UTC m=+411.839865309" Oct 30 19:24:02.618457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.618427 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg"] Oct 30 19:24:02.621136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.621119 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.642194 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.642170 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 19:24:02.642506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.642391 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg"] Oct 30 19:24:02.697589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.697566 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:02.721375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.721346 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:02.729618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.729588 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dda9c25-033d-4e68-bcf8-ba726c31024e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.729703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.729652 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.729758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.729737 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.729827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.729791 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.729901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.729831 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fq89\" (UniqueName: \"kubernetes.io/projected/6dda9c25-033d-4e68-bcf8-ba726c31024e-kube-api-access-9fq89\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.729901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.729887 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.830901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.830852 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dda9c25-033d-4e68-bcf8-ba726c31024e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.830934 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.830974 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.830998 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.831035 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fq89\" (UniqueName: \"kubernetes.io/projected/6dda9c25-033d-4e68-bcf8-ba726c31024e-kube-api-access-9fq89\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.831073 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.831367 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.831408 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.831484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.831462 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.833289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.833263 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dda9c25-033d-4e68-bcf8-ba726c31024e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.833455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.833436 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dda9c25-033d-4e68-bcf8-ba726c31024e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.841417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.841393 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fq89\" (UniqueName: \"kubernetes.io/projected/6dda9c25-033d-4e68-bcf8-ba726c31024e-kube-api-access-9fq89\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg\" (UID: \"6dda9c25-033d-4e68-bcf8-ba726c31024e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:02.932701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:02.932680 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:03.242200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:03.242152 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg"] Oct 30 19:24:03.244327 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:24:03.244298 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dda9c25_033d_4e68_bcf8_ba726c31024e.slice/crio-a989a4286e2738a2df8307f510a40f5073666ace3e781bcefbc5302c29d5c433 WatchSource:0}: Error finding container a989a4286e2738a2df8307f510a40f5073666ace3e781bcefbc5302c29d5c433: Status 404 returned error can't find the container with id a989a4286e2738a2df8307f510a40f5073666ace3e781bcefbc5302c29d5c433 Oct 30 19:24:03.529173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:03.529148 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:03.872543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:03.872461 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" event={"ID":"6dda9c25-033d-4e68-bcf8-ba726c31024e","Type":"ContainerStarted","Data":"67ddbe81a6f0ad2642966522a722c512beeb8087bc052bc47dbdc00dbe98db72"} Oct 30 19:24:03.872543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:03.872525 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" event={"ID":"6dda9c25-033d-4e68-bcf8-ba726c31024e","Type":"ContainerStarted","Data":"a989a4286e2738a2df8307f510a40f5073666ace3e781bcefbc5302c29d5c433"} Oct 30 19:24:03.901456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:03.901414 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" podStartSLOduration=1.63987106 podStartE2EDuration="1.901401425s" totalImagesPullingTime="261.530365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:03.897895052 +0000 UTC m=+413.855714236" watchObservedRunningTime="2026-10-30 19:24:03.901401425 +0000 UTC m=+413.859220613" Oct 30 19:24:03.933560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:03.933537 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:04.362622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:04.362589 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:24:04.375130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:04.375106 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:24:04.887552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:04.887523 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" Oct 30 19:24:05.714343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:05.714311 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:24:05.726496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:05.726468 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:24:05.891817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:05.891792 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:24:06.802375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:06.802326 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:24:07.264558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:07.264518 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:24:07.276906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:07.276882 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:24:07.901323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:07.901298 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" Oct 30 19:24:09.775719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:09.775677 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:10.697596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:10.697557 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:10.710060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:10.710037 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:10.916141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:10.916116 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" Oct 30 19:24:12.933092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:12.933061 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:12.945271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:12.945242 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:13.929115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:13.929087 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" Oct 30 19:24:18.930392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.930361 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-gpcsx"] Oct 30 19:24:18.933255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.933234 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:18.948493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.948470 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 19:24:18.948586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.948543 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-kltdj\"" Oct 30 19:24:18.949614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.949590 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-gpcsx"] Oct 30 19:24:18.949805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.949782 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:24:18.957397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.957367 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs\") pod \"ai-gateway-controller-655fc879db-gpcsx\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:18.957538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.957457 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsn4r\" (UniqueName: \"kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r\") pod \"ai-gateway-controller-655fc879db-gpcsx\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:18.959262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:18.959237 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:19.058382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.058357 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsn4r\" (UniqueName: \"kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r\") pod \"ai-gateway-controller-655fc879db-gpcsx\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:19.058493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.058455 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs\") pod \"ai-gateway-controller-655fc879db-gpcsx\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:19.060777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.060757 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs\") pod \"ai-gateway-controller-655fc879db-gpcsx\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:19.066995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.066969 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsn4r\" (UniqueName: \"kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r\") pod \"ai-gateway-controller-655fc879db-gpcsx\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:19.076180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.076129 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:19.244691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.244619 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:19.545525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.545451 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-gpcsx"] Oct 30 19:24:19.549293 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:24:19.549262 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbdb74b_d7b8_468f_abc6_4d1e00659a06.slice/crio-1f5bd78ae0e92d0f62ddcd31b5fe40ba8fb8548442987e75f94ab0d01a7e7e48 WatchSource:0}: Error finding container 1f5bd78ae0e92d0f62ddcd31b5fe40ba8fb8548442987e75f94ab0d01a7e7e48: Status 404 returned error can't find the container with id 1f5bd78ae0e92d0f62ddcd31b5fe40ba8fb8548442987e75f94ab0d01a7e7e48 Oct 30 19:24:19.947345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:19.947314 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" event={"ID":"1fbdb74b-d7b8-468f-abc6-4d1e00659a06","Type":"ContainerStarted","Data":"1f5bd78ae0e92d0f62ddcd31b5fe40ba8fb8548442987e75f94ab0d01a7e7e48"} Oct 30 19:24:20.880143 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:24:20.880105 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:24:23.972918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:23.972884 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" event={"ID":"1fbdb74b-d7b8-468f-abc6-4d1e00659a06","Type":"ContainerStarted","Data":"8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1"} Oct 30 19:24:23.973296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:23.973126 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:23.992579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:23.992533 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" podStartSLOduration=2.363906882 podStartE2EDuration="5.992520222s" totalImagesPullingTime="3.62861334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:23.991242851 +0000 UTC m=+433.949062063" watchObservedRunningTime="2026-10-30 19:24:23.992520222 +0000 UTC m=+433.950339410" Oct 30 19:24:25.084126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.084092 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:25.141665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.141636 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh"] Oct 30 19:24:25.145249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.145233 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.157781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.157762 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:24:25.157781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.157773 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g9c8m\"" Oct 30 19:24:25.168693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.168664 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh"] Oct 30 19:24:25.206042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.206014 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04daab4c-1b99-4724-8649-630cdb2b4bc4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v5vhh\" (UID: \"04daab4c-1b99-4724-8649-630cdb2b4bc4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.206141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.206053 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzj4r\" (UniqueName: \"kubernetes.io/projected/04daab4c-1b99-4724-8649-630cdb2b4bc4-kube-api-access-mzj4r\") pod \"payload-pre-processing-7b98bb678c-v5vhh\" (UID: \"04daab4c-1b99-4724-8649-630cdb2b4bc4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.276033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.276002 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:25.307360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.307331 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04daab4c-1b99-4724-8649-630cdb2b4bc4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v5vhh\" (UID: \"04daab4c-1b99-4724-8649-630cdb2b4bc4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.307501 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.307371 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzj4r\" (UniqueName: \"kubernetes.io/projected/04daab4c-1b99-4724-8649-630cdb2b4bc4-kube-api-access-mzj4r\") pod \"payload-pre-processing-7b98bb678c-v5vhh\" (UID: \"04daab4c-1b99-4724-8649-630cdb2b4bc4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.308096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.308076 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04daab4c-1b99-4724-8649-630cdb2b4bc4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v5vhh\" (UID: \"04daab4c-1b99-4724-8649-630cdb2b4bc4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.316558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.316536 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzj4r\" (UniqueName: \"kubernetes.io/projected/04daab4c-1b99-4724-8649-630cdb2b4bc4-kube-api-access-mzj4r\") pod \"payload-pre-processing-7b98bb678c-v5vhh\" (UID: \"04daab4c-1b99-4724-8649-630cdb2b4bc4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.457208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.457182 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:25.735969 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:24:25.735930 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04daab4c_1b99_4724_8649_630cdb2b4bc4.slice/crio-e7dc822cf842f1e54036650b3baeae212f25cc7ebb4e359822006e56be9218b7 WatchSource:0}: Error finding container e7dc822cf842f1e54036650b3baeae212f25cc7ebb4e359822006e56be9218b7: Status 404 returned error can't find the container with id e7dc822cf842f1e54036650b3baeae212f25cc7ebb4e359822006e56be9218b7 Oct 30 19:24:25.736410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.736390 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh"] Oct 30 19:24:25.983361 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:25.983280 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" event={"ID":"04daab4c-1b99-4724-8649-630cdb2b4bc4","Type":"ContainerStarted","Data":"e7dc822cf842f1e54036650b3baeae212f25cc7ebb4e359822006e56be9218b7"} Oct 30 19:24:27.995051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:27.995015 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" event={"ID":"04daab4c-1b99-4724-8649-630cdb2b4bc4","Type":"ContainerStarted","Data":"334898f0c031156d16acbce549493a4a31bed5fd58e37b788d044281c5293c3b"} Oct 30 19:24:27.995439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:27.995193 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:28.017410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:28.017366 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" podStartSLOduration=0.969866151 podStartE2EDuration="3.017353837s" totalImagesPullingTime="2.047487686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:28.014447774 +0000 UTC m=+437.972266960" watchObservedRunningTime="2026-10-30 19:24:28.017353837 +0000 UTC m=+437.975173032" Oct 30 19:24:34.979953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:34.979926 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:35.268719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.268637 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2"] Oct 30 19:24:35.274322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.274304 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.281795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.281775 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2"] Oct 30 19:24:35.394044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.394021 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ff1115-85eb-4812-a46a-5bf2b3582607-metrics-certs\") pod \"ai-gateway-controller-7d59c5fdcf-dmjs2\" (UID: \"d1ff1115-85eb-4812-a46a-5bf2b3582607\") " pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.394136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.394053 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxjj\" (UniqueName: \"kubernetes.io/projected/d1ff1115-85eb-4812-a46a-5bf2b3582607-kube-api-access-ddxjj\") pod \"ai-gateway-controller-7d59c5fdcf-dmjs2\" (UID: \"d1ff1115-85eb-4812-a46a-5bf2b3582607\") " pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.495192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.495165 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ff1115-85eb-4812-a46a-5bf2b3582607-metrics-certs\") pod \"ai-gateway-controller-7d59c5fdcf-dmjs2\" (UID: \"d1ff1115-85eb-4812-a46a-5bf2b3582607\") " pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.495298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.495199 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxjj\" (UniqueName: \"kubernetes.io/projected/d1ff1115-85eb-4812-a46a-5bf2b3582607-kube-api-access-ddxjj\") pod \"ai-gateway-controller-7d59c5fdcf-dmjs2\" (UID: \"d1ff1115-85eb-4812-a46a-5bf2b3582607\") " pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.497629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.497606 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ff1115-85eb-4812-a46a-5bf2b3582607-metrics-certs\") pod \"ai-gateway-controller-7d59c5fdcf-dmjs2\" (UID: \"d1ff1115-85eb-4812-a46a-5bf2b3582607\") " pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.505191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.505169 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxjj\" (UniqueName: \"kubernetes.io/projected/d1ff1115-85eb-4812-a46a-5bf2b3582607-kube-api-access-ddxjj\") pod \"ai-gateway-controller-7d59c5fdcf-dmjs2\" (UID: \"d1ff1115-85eb-4812-a46a-5bf2b3582607\") " pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.587335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.587286 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:35.860082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:35.860051 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2"] Oct 30 19:24:35.861199 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:24:35.861171 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ff1115_85eb_4812_a46a_5bf2b3582607.slice/crio-95edd06696c5c9324dddd3656553c3dcc61907f759cfe5909c072c3281e04044 WatchSource:0}: Error finding container 95edd06696c5c9324dddd3656553c3dcc61907f759cfe5909c072c3281e04044: Status 404 returned error can't find the container with id 95edd06696c5c9324dddd3656553c3dcc61907f759cfe5909c072c3281e04044 Oct 30 19:24:36.031493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:36.031455 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" event={"ID":"d1ff1115-85eb-4812-a46a-5bf2b3582607","Type":"ContainerStarted","Data":"95edd06696c5c9324dddd3656553c3dcc61907f759cfe5909c072c3281e04044"} Oct 30 19:24:37.037233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:37.037192 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" event={"ID":"d1ff1115-85eb-4812-a46a-5bf2b3582607","Type":"ContainerStarted","Data":"dfb1d8365b6d5632be4b147f0d949b9a32839a62bd99e36c303bf7baf4ef4341"} Oct 30 19:24:37.037743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:37.037269 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:37.057102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:37.057053 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" podStartSLOduration=1.6577237089999999 podStartE2EDuration="2.057040461s" totalImagesPullingTime="399.316752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:37.056748572 +0000 UTC m=+447.014567761" watchObservedRunningTime="2026-10-30 19:24:37.057040461 +0000 UTC m=+447.014859649" Oct 30 19:24:39.003376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:39.003347 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" Oct 30 19:24:48.044046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.043969 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d59c5fdcf-dmjs2" Oct 30 19:24:48.099547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.099514 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-gpcsx"] Oct 30 19:24:48.099754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.099719 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" podUID="1fbdb74b-d7b8-468f-abc6-4d1e00659a06" containerName="manager" containerID="cri-o://8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1" gracePeriod=30 Oct 30 19:24:48.331600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.331578 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:48.398110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.398078 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs\") pod \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " Oct 30 19:24:48.398230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.398143 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r\" (UniqueName: \"kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r\") pod \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\" (UID: \"1fbdb74b-d7b8-468f-abc6-4d1e00659a06\") " Oct 30 19:24:48.400125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.400100 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs" pod "1fbdb74b-d7b8-468f-abc6-4d1e00659a06" (UID: "1fbdb74b-d7b8-468f-abc6-4d1e00659a06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:24:48.400237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.400191 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r" pod "1fbdb74b-d7b8-468f-abc6-4d1e00659a06" (UID: "1fbdb74b-d7b8-468f-abc6-4d1e00659a06"). InnerVolumeSpecName "kube-api-access-nsn4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:48.499392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.499367 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:48.499392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:48.499390 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsn4r\" (UniqueName: \"kubernetes.io/projected/1fbdb74b-d7b8-468f-abc6-4d1e00659a06-kube-api-access-nsn4r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:49.092013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.091990 6514 generic.go:356] "Generic (PLEG): container finished" podID="1fbdb74b-d7b8-468f-abc6-4d1e00659a06" containerID="8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1" exitCode=0 Oct 30 19:24:49.092382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.092030 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" Oct 30 19:24:49.092382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.092078 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" event={"ID":"1fbdb74b-d7b8-468f-abc6-4d1e00659a06","Type":"ContainerDied","Data":"8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1"} Oct 30 19:24:49.092382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.092130 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-gpcsx" event={"ID":"1fbdb74b-d7b8-468f-abc6-4d1e00659a06","Type":"ContainerDied","Data":"1f5bd78ae0e92d0f62ddcd31b5fe40ba8fb8548442987e75f94ab0d01a7e7e48"} Oct 30 19:24:49.092382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.092155 6514 scope.go:122] "RemoveContainer" containerID="8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1" Oct 30 19:24:49.101509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.101360 6514 scope.go:122] "RemoveContainer" containerID="8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1" Oct 30 19:24:49.101633 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:24:49.101617 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1\": container with ID starting with 8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1 not found: ID does not exist" containerID="8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1" Oct 30 19:24:49.101675 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.101640 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1"} err="failed to get container status \"8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1\": rpc error: code = NotFound desc = could not find container \"8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1\": container with ID starting with 8e7c32442d3280593813b24f6794822dcfb65cfb2715ad597e1a0e9d63cb0df1 not found: ID does not exist" Oct 30 19:24:49.115218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.115197 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-gpcsx"] Oct 30 19:24:49.120354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.120331 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-gpcsx"] Oct 30 19:24:49.896351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:49.896325 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:50.156763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:50.156700 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbdb74b-d7b8-468f-abc6-4d1e00659a06" path="/var/lib/kubelet/pods/1fbdb74b-d7b8-468f-abc6-4d1e00659a06/volumes" Oct 30 19:24:50.407444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:50.407371 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:50.879291 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:24:50.879252 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:24:52.115853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:52.115822 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:52.236770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:52.236737 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:53.632448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:53.632412 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:53.893025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:24:53.892943 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:09.819396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:09.819364 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:20.879333 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:25:20.879294 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:25:25.605576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:25.605538 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:25.737516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:25.737480 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.129391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:26.129349 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.298395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:26.298364 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.603156 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:26.603126 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.777209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:26.777181 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.926342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:26.926267 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:27.060129 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:27.060084 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:29.861000 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:29.860972 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.047349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:30.047314 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.181531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:30.181495 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.345941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:30.345911 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.669182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:30.669109 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.838409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:30.838379 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:31.022021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.021991 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:31.156877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.156845 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:31.340012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.339939 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:31.371685 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.371655 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc"] Oct 30 19:25:31.372095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.372078 6514 cpu_manager.go:414] "containerMap: removing container" podUID="1fbdb74b-d7b8-468f-abc6-4d1e00659a06" containerName="manager" Oct 30 19:25:31.372152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.372098 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbdb74b-d7b8-468f-abc6-4d1e00659a06" containerName="manager" Oct 30 19:25:31.372192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.372170 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbdb74b-d7b8-468f-abc6-4d1e00659a06" containerName="manager" Oct 30 19:25:31.374934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.374917 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.385090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.385071 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ce69f3-dockercfg-c7kfb\"" Oct 30 19:25:31.388537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.388517 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ce69f3\"" Oct 30 19:25:31.391172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.391147 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc"] Oct 30 19:25:31.419098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.419074 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw"] Oct 30 19:25:31.422361 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.422346 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.432633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.432613 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk62p\" (UniqueName: \"kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p\") pod \"payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.432730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.432673 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58gdz\" (UniqueName: \"kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz\") pod \"payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.432781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.432733 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.432844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.432824 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.441841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.441823 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw"] Oct 30 19:25:31.482412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.482387 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:31.534095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.534029 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk62p\" (UniqueName: \"kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p\") pod \"payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.534566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.534540 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58gdz\" (UniqueName: \"kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz\") pod \"payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.534813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.534601 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.534813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.534649 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.535615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.535587 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.535758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.535596 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.557175 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.557151 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk62p\" (UniqueName: \"kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p\") pod \"payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.557487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.557470 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58gdz\" (UniqueName: \"kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz\") pod \"payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.688182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.688155 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:31.734478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.734448 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:31.965570 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:31.965545 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc"] Oct 30 19:25:31.967513 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:31.967488 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428a3c12_c612_40b0_a039_b08536c54541.slice/crio-7fdfdc7f868deec844affed62d1adcbd54630ef2bb697fd9cf24f5dea0ce79e7 WatchSource:0}: Error finding container 7fdfdc7f868deec844affed62d1adcbd54630ef2bb697fd9cf24f5dea0ce79e7: Status 404 returned error can't find the container with id 7fdfdc7f868deec844affed62d1adcbd54630ef2bb697fd9cf24f5dea0ce79e7 Oct 30 19:25:32.019653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.019593 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw"] Oct 30 19:25:32.021398 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:32.021377 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7802d20_9ba2_475f_8d33_05497773ba90.slice/crio-3ec09ac25b7ecd8c8d5161dcaf28f4b85939b967f361d8c15b8a2d574f3af889 WatchSource:0}: Error finding container 3ec09ac25b7ecd8c8d5161dcaf28f4b85939b967f361d8c15b8a2d574f3af889: Status 404 returned error can't find the container with id 3ec09ac25b7ecd8c8d5161dcaf28f4b85939b967f361d8c15b8a2d574f3af889 Oct 30 19:25:32.274207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.274126 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt"] Oct 30 19:25:32.277613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.277591 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.283391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.283362 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" event={"ID":"428a3c12-c612-40b0-a039-b08536c54541","Type":"ContainerStarted","Data":"0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127"} Oct 30 19:25:32.283527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.283408 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" event={"ID":"428a3c12-c612-40b0-a039-b08536c54541","Type":"ContainerStarted","Data":"7fdfdc7f868deec844affed62d1adcbd54630ef2bb697fd9cf24f5dea0ce79e7"} Oct 30 19:25:32.283527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.283475 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:32.285052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.285025 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" event={"ID":"f7802d20-9ba2-475f-8d33-05497773ba90","Type":"ContainerStarted","Data":"09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18"} Oct 30 19:25:32.285171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.285067 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" event={"ID":"f7802d20-9ba2-475f-8d33-05497773ba90","Type":"ContainerStarted","Data":"3ec09ac25b7ecd8c8d5161dcaf28f4b85939b967f361d8c15b8a2d574f3af889"} Oct 30 19:25:32.285171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.285109 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:32.289260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.289230 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:25:32.289509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.289498 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ce69f3-serving-cert\"" Oct 30 19:25:32.289693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.289679 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2z7lq\"" Oct 30 19:25:32.289738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.289678 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:25:32.289738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.289704 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:25:32.293779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.293757 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt"] Oct 30 19:25:32.341151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.341128 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.341294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.341165 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.341294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.341196 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.341759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.341378 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwn9\" (UniqueName: \"kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.346632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.346594 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" podStartSLOduration=1.346582363 podStartE2EDuration="1.346582363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:32.345628491 +0000 UTC m=+502.303447676" watchObservedRunningTime="2026-10-30 19:25:32.346582363 +0000 UTC m=+502.304401580" Oct 30 19:25:32.366686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.366653 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:32.372199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.372159 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" podStartSLOduration=1.372147044 podStartE2EDuration="1.372147044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:32.370685934 +0000 UTC m=+502.328505123" watchObservedRunningTime="2026-10-30 19:25:32.372147044 +0000 UTC m=+502.329966234" Oct 30 19:25:32.442182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.442159 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.442182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.442188 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.442333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.442206 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.442333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.442257 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwn9\" (UniqueName: \"kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.442772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.442749 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.444701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.444682 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.444779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.444687 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.451936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.451917 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwn9\" (UniqueName: \"kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9\") pod \"maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.471762 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.471738 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6"] Oct 30 19:25:32.475477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.475462 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.483109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.483087 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-01b24e-dockercfg-grp9v\"" Oct 30 19:25:32.483199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.483175 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-01b24e\"" Oct 30 19:25:32.485350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.485329 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6"] Oct 30 19:25:32.492731 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.492711 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:32.499767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.499744 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph"] Oct 30 19:25:32.503917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.503899 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.512009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.511987 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph"] Oct 30 19:25:32.544217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.544138 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.544329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.544215 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfr8n\" (UniqueName: \"kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n\") pod \"payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.544329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.544313 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.544445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.544382 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcpb\" (UniqueName: \"kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb\") pod \"payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.591250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.591223 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:32.645526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.645496 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfr8n\" (UniqueName: \"kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n\") pod \"payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.645632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.645550 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.645632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.645611 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcpb\" (UniqueName: \"kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb\") pod \"payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.645748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.645725 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.646513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.646490 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.646734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.646710 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.660892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.656094 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcpb\" (UniqueName: \"kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb\") pod \"payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.660892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.656617 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfr8n\" (UniqueName: \"kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n\") pod \"payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.788611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.788577 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:32.816745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.816718 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:32.820752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.820723 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:32.872961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.872926 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs"] Oct 30 19:25:32.876155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.876131 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:25:32.878364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.878159 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt"] Oct 30 19:25:32.878364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.878260 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:32.894372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.887704 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-60577a-dockercfg-7td8n\"" Oct 30 19:25:32.894372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.888564 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-60577a\"" Oct 30 19:25:32.895981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.895959 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs"] Oct 30 19:25:32.923957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.923933 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j"] Oct 30 19:25:32.927938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.927916 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:32.940779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.940752 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j"] Oct 30 19:25:32.948475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.948453 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:32.948567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.948498 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42fn\" (UniqueName: \"kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn\") pod \"payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:32.948620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.948599 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsb8q\" (UniqueName: \"kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q\") pod \"payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:32.948758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.948737 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:32.951522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:32.951497 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:33.049930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.049849 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42fn\" (UniqueName: \"kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn\") pod \"payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:33.050059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.049930 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb8q\" (UniqueName: \"kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q\") pod \"payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:33.050059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.050018 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:33.050145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.050074 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:33.050684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.050663 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:33.050724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.050678 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:33.062349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.062324 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42fn\" (UniqueName: \"kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn\") pod \"payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:33.063630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.063613 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsb8q\" (UniqueName: \"kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q\") pod \"payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:33.074846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.074822 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6"] Oct 30 19:25:33.076443 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:33.076420 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15ed13e_ac83_4499_b154_05c89333c29c.slice/crio-fc67856103a9adf9ba8e03d52c3ed6494d57b19a1e518d1d83d23da192ff8ffa WatchSource:0}: Error finding container fc67856103a9adf9ba8e03d52c3ed6494d57b19a1e518d1d83d23da192ff8ffa: Status 404 returned error can't find the container with id fc67856103a9adf9ba8e03d52c3ed6494d57b19a1e518d1d83d23da192ff8ffa Oct 30 19:25:33.119619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.119599 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph"] Oct 30 19:25:33.121598 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:33.121576 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269a3f6e_538b_4be8_8a51_b056342c28fe.slice/crio-9b8c12a60dfd332def751a3149fcb1e008eabe91d21c95ad1c0bf5267a166e4c WatchSource:0}: Error finding container 9b8c12a60dfd332def751a3149fcb1e008eabe91d21c95ad1c0bf5267a166e4c: Status 404 returned error can't find the container with id 9b8c12a60dfd332def751a3149fcb1e008eabe91d21c95ad1c0bf5267a166e4c Oct 30 19:25:33.212443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.212420 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:33.244474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.244450 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:33.297941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.296956 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" event={"ID":"269a3f6e-538b-4be8-8a51-b056342c28fe","Type":"ContainerStarted","Data":"665929b2a5feff16b62cfab37f681fb1c69ba6d45fe06a456121f3ebab986783"} Oct 30 19:25:33.297941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.297022 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" event={"ID":"269a3f6e-538b-4be8-8a51-b056342c28fe","Type":"ContainerStarted","Data":"9b8c12a60dfd332def751a3149fcb1e008eabe91d21c95ad1c0bf5267a166e4c"} Oct 30 19:25:33.297941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.297852 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:33.300245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.300165 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" event={"ID":"e15ed13e-ac83-4499-b154-05c89333c29c","Type":"ContainerStarted","Data":"79b5160541ecbb181427132ebce23f45b173467f6d4683dffe498f4d24be5895"} Oct 30 19:25:33.300245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.300231 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" event={"ID":"e15ed13e-ac83-4499-b154-05c89333c29c","Type":"ContainerStarted","Data":"fc67856103a9adf9ba8e03d52c3ed6494d57b19a1e518d1d83d23da192ff8ffa"} Oct 30 19:25:33.300689 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.300673 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:33.302990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.302972 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" event={"ID":"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a","Type":"ContainerStarted","Data":"a553cbc0905590179f1e2ab596d0594b8065e365dc9ace913dc6ca7c88fd4011"} Oct 30 19:25:33.332626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.332580 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" podStartSLOduration=1.332566466 podStartE2EDuration="1.332566466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:33.331132945 +0000 UTC m=+503.288952135" watchObservedRunningTime="2026-10-30 19:25:33.332566466 +0000 UTC m=+503.290385653" Oct 30 19:25:33.364187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.364144 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" podStartSLOduration=1.364131475 podStartE2EDuration="1.364131475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:33.36270974 +0000 UTC m=+503.320528928" watchObservedRunningTime="2026-10-30 19:25:33.364131475 +0000 UTC m=+503.321950692" Oct 30 19:25:33.491052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.490995 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:33.497460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.497435 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs"] Oct 30 19:25:33.500294 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:33.500271 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd3059d_f277_495c_b6a9_8fea7f88eb2f.slice/crio-38216830188d786eb495111bb38949b9640bd375cbf5d99b7a01a39209a25609 WatchSource:0}: Error finding container 38216830188d786eb495111bb38949b9640bd375cbf5d99b7a01a39209a25609: Status 404 returned error can't find the container with id 38216830188d786eb495111bb38949b9640bd375cbf5d99b7a01a39209a25609 Oct 30 19:25:33.538433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:33.538405 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j"] Oct 30 19:25:33.539791 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:33.539771 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2d179b_367b_4a85_9044_e9edb74fd53b.slice/crio-1f38f9f640ade649bbe8f6f6b4547f67ef7fc50aedb972e17be8ccee5c760271 WatchSource:0}: Error finding container 1f38f9f640ade649bbe8f6f6b4547f67ef7fc50aedb972e17be8ccee5c760271: Status 404 returned error can't find the container with id 1f38f9f640ade649bbe8f6f6b4547f67ef7fc50aedb972e17be8ccee5c760271 Oct 30 19:25:34.046240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.046208 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.072687 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.072658 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j"] Oct 30 19:25:34.076534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.076512 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.089688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.089643 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-419d0a\"" Oct 30 19:25:34.092146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.092102 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-419d0a-dockercfg-j7vgb\"" Oct 30 19:25:34.094270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.094242 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j"] Oct 30 19:25:34.140528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.140501 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl"] Oct 30 19:25:34.143964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.143947 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.160985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.160957 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl"] Oct 30 19:25:34.161245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.161222 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.161305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.161291 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77t8q\" (UniqueName: \"kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q\") pod \"payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.161369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.161357 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.161410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.161393 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vcs\" (UniqueName: \"kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs\") pod \"payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.198292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.198263 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.262562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.262522 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vcs\" (UniqueName: \"kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs\") pod \"payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.262711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.262588 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.262711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.262621 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77t8q\" (UniqueName: \"kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q\") pod \"payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.262711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.262672 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.263369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.263350 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.263406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.263378 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.271985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.271958 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77t8q\" (UniqueName: \"kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q\") pod \"payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.272126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.272068 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vcs\" (UniqueName: \"kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs\") pod \"payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.311575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.311503 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" event={"ID":"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a","Type":"ContainerStarted","Data":"a28bd39fa4d3fe20eb2ebc81ca6412d9ae545ff8dd0e139038791d60c44c608a"} Oct 30 19:25:34.311575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.311559 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:34.313331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.313302 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" event={"ID":"6b2d179b-367b-4a85-9044-e9edb74fd53b","Type":"ContainerStarted","Data":"d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026"} Oct 30 19:25:34.313417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.313347 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" event={"ID":"6b2d179b-367b-4a85-9044-e9edb74fd53b","Type":"ContainerStarted","Data":"1f38f9f640ade649bbe8f6f6b4547f67ef7fc50aedb972e17be8ccee5c760271"} Oct 30 19:25:34.313417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.313367 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:34.314903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.314885 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" event={"ID":"0fd3059d-f277-495c-b6a9-8fea7f88eb2f","Type":"ContainerStarted","Data":"eb387434a26e500a04e9ffbc11b5737d77be559f84898e3d0a101d97af5e7a33"} Oct 30 19:25:34.314991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.314924 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" event={"ID":"0fd3059d-f277-495c-b6a9-8fea7f88eb2f","Type":"ContainerStarted","Data":"38216830188d786eb495111bb38949b9640bd375cbf5d99b7a01a39209a25609"} Oct 30 19:25:34.333414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.333379 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" podStartSLOduration=1.831934846 podStartE2EDuration="2.333366096s" totalImagesPullingTime="501.43125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:34.332530792 +0000 UTC m=+504.290349991" watchObservedRunningTime="2026-10-30 19:25:34.333366096 +0000 UTC m=+504.291185285" Oct 30 19:25:34.356679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.356642 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" podStartSLOduration=2.356632421 podStartE2EDuration="2.356632421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:34.35556471 +0000 UTC m=+504.313383898" watchObservedRunningTime="2026-10-30 19:25:34.356632421 +0000 UTC m=+504.314451608" Oct 30 19:25:34.376064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.376039 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g"] Oct 30 19:25:34.380651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.380610 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" podStartSLOduration=2.3805973959999998 podStartE2EDuration="2.380597396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:34.378796142 +0000 UTC m=+504.336615322" watchObservedRunningTime="2026-10-30 19:25:34.380597396 +0000 UTC m=+504.338416584" Oct 30 19:25:34.381091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.381075 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.389186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.389146 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-01b24e-serving-cert\"" Oct 30 19:25:34.390616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.390461 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:34.391196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.391172 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g"] Oct 30 19:25:34.457221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.457192 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:34.458711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.458687 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.464701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.464675 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nxv\" (UniqueName: \"kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.464790 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.464723 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.464856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.464792 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.464856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.464842 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.566135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.565993 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nxv\" (UniqueName: \"kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.566135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.566056 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.566135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.566099 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.566369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.566141 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.566921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.566895 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.568488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.568467 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.568594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.568543 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.577680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.577659 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nxv\" (UniqueName: \"kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv\") pod \"maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.581906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.581880 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.671981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.671959 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j"] Oct 30 19:25:34.675896 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:34.675242 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19b0962_4976_45a6_bffa_06720cab3b25.slice/crio-688181f22f1d6f0e393ea718fe177927b3c2f1073cce8dca5eee20576bb75085 WatchSource:0}: Error finding container 688181f22f1d6f0e393ea718fe177927b3c2f1073cce8dca5eee20576bb75085: Status 404 returned error can't find the container with id 688181f22f1d6f0e393ea718fe177927b3c2f1073cce8dca5eee20576bb75085 Oct 30 19:25:34.699711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.699690 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:34.717485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.717454 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.744202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.744167 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl"] Oct 30 19:25:34.745014 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:34.744986 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d975d56_a1b1_4cdf_9dd1_3691ccabafbe.slice/crio-e06e87b1f4cea7a8e2de22eab2c8992bcfa435e9c86fe8e459771d1def45329b WatchSource:0}: Error finding container e06e87b1f4cea7a8e2de22eab2c8992bcfa435e9c86fe8e459771d1def45329b: Status 404 returned error can't find the container with id e06e87b1f4cea7a8e2de22eab2c8992bcfa435e9c86fe8e459771d1def45329b Oct 30 19:25:34.804857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.804831 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh"] Oct 30 19:25:34.809315 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.809295 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.822296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.822205 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b623ca\"" Oct 30 19:25:34.822538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.822519 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b623ca-dockercfg-f5sbg\"" Oct 30 19:25:34.823629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.823609 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh"] Oct 30 19:25:34.839463 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.839439 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44"] Oct 30 19:25:34.842912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.842897 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.857889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.856983 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44"] Oct 30 19:25:34.866355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.866334 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.868613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.868597 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47z7\" (UniqueName: \"kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7\") pod \"payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.868674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.868646 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fp2p\" (UniqueName: \"kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p\") pod \"payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.868753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.868736 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.868787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.868767 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.970139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.970109 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.970304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.970211 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47z7\" (UniqueName: \"kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7\") pod \"payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.970388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.970368 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fp2p\" (UniqueName: \"kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p\") pod \"payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.970446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.970433 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.971151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.971129 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.971151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.971132 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.979152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.979127 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fp2p\" (UniqueName: \"kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p\") pod \"payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:34.979294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.979278 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47z7\" (UniqueName: \"kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7\") pod \"payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:34.985413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:34.985392 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g"] Oct 30 19:25:34.987207 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:34.987187 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc241c2_1681_4058_97ea_980111f33d4b.slice/crio-c109176a2590d8b9881523d1259159d88c667639af38de3ff4ccc994a9f0ff7f WatchSource:0}: Error finding container c109176a2590d8b9881523d1259159d88c667639af38de3ff4ccc994a9f0ff7f: Status 404 returned error can't find the container with id c109176a2590d8b9881523d1259159d88c667639af38de3ff4ccc994a9f0ff7f Oct 30 19:25:35.122630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.122569 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:35.155597 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.155572 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:35.322505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.322467 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" event={"ID":"e19b0962-4976-45a6-bffa-06720cab3b25","Type":"ContainerStarted","Data":"9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467"} Oct 30 19:25:35.322917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.322534 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" event={"ID":"e19b0962-4976-45a6-bffa-06720cab3b25","Type":"ContainerStarted","Data":"688181f22f1d6f0e393ea718fe177927b3c2f1073cce8dca5eee20576bb75085"} Oct 30 19:25:35.322917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.322582 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:35.324027 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.323999 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" event={"ID":"abc241c2-1681-4058-97ea-980111f33d4b","Type":"ContainerStarted","Data":"c109176a2590d8b9881523d1259159d88c667639af38de3ff4ccc994a9f0ff7f"} Oct 30 19:25:35.325641 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.325617 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" event={"ID":"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe","Type":"ContainerStarted","Data":"0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74"} Oct 30 19:25:35.325764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.325661 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" event={"ID":"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe","Type":"ContainerStarted","Data":"e06e87b1f4cea7a8e2de22eab2c8992bcfa435e9c86fe8e459771d1def45329b"} Oct 30 19:25:35.325764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.325690 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:35.325892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.325881 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:35.350211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.350181 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9"] Oct 30 19:25:35.352908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.352848 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" podStartSLOduration=1.352836988 podStartE2EDuration="1.352836988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:35.350692145 +0000 UTC m=+505.308511332" watchObservedRunningTime="2026-10-30 19:25:35.352836988 +0000 UTC m=+505.310656176" Oct 30 19:25:35.354075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.354057 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.356186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.356165 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.365138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.365115 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9"] Oct 30 19:25:35.366599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.366579 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ad96a4-dockercfg-lckc4\"" Oct 30 19:25:35.366768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.366754 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ad96a4\"" Oct 30 19:25:35.374415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.374349 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.374518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.374494 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgz2\" (UniqueName: \"kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2\") pod \"payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.377376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.377335 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" podStartSLOduration=1.377323562 podStartE2EDuration="1.377323562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:35.375320059 +0000 UTC m=+505.333139261" watchObservedRunningTime="2026-10-30 19:25:35.377323562 +0000 UTC m=+505.335142746" Oct 30 19:25:35.410103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.410084 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh"] Oct 30 19:25:35.417529 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:35.417500 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01ce9d1_f33c_476a_bd2d_ed66b3b9e2b9.slice/crio-40690d054d9ba1123a9f7ff3bae6dd83a62623efd156395f7857e29d3666c4f8 WatchSource:0}: Error finding container 40690d054d9ba1123a9f7ff3bae6dd83a62623efd156395f7857e29d3666c4f8: Status 404 returned error can't find the container with id 40690d054d9ba1123a9f7ff3bae6dd83a62623efd156395f7857e29d3666c4f8 Oct 30 19:25:35.443972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.443944 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44"] Oct 30 19:25:35.445829 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:35.445809 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92245d2_8752_4350_9266_dac03282e97c.slice/crio-82d8babdd38ed54d654dda917a3cefcd604b6010e4e16f2224514d0f3c9b6c12 WatchSource:0}: Error finding container 82d8babdd38ed54d654dda917a3cefcd604b6010e4e16f2224514d0f3c9b6c12: Status 404 returned error can't find the container with id 82d8babdd38ed54d654dda917a3cefcd604b6010e4e16f2224514d0f3c9b6c12 Oct 30 19:25:35.476407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.476380 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.476511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.476454 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgz2\" (UniqueName: \"kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2\") pod \"payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.478295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.477273 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.478295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.477750 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:25:35.489065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.488846 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgz2\" (UniqueName: \"kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2\") pod \"payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.504284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.504256 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.592433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.590085 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj"] Oct 30 19:25:35.594803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.594778 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.610662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.608110 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-60577a-serving-cert\"" Oct 30 19:25:35.610662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.610549 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj"] Oct 30 19:25:35.670874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.670790 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:35.673109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.673082 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.678980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.678945 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.679108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.679086 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.679178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.679122 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.679230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.679191 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7dp\" (UniqueName: \"kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.780657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.780627 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.780657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.780659 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.780889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.780678 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7dp\" (UniqueName: \"kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.780889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.780760 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.780889 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:25:35.780780 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-serving-cert: secret "maas-api-e2e-worker-w5-60577a-serving-cert" not found Oct 30 19:25:35.780889 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:25:35.780852 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls podName:8930e474-eb64-4d26-abe4-45d6f1d9c0df nodeName:}" failed. No retries permitted until 2026-10-30 19:25:36.28083557 +0000 UTC m=+506.238654740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls") pod "maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" (UID: "8930e474-eb64-4d26-abe4-45d6f1d9c0df") : secret "maas-api-e2e-worker-w5-60577a-serving-cert" not found Oct 30 19:25:35.781475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.781456 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.783083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.783065 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.786187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.786165 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.793105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.793084 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7dp\" (UniqueName: \"kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:35.956629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:35.956533 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9"] Oct 30 19:25:35.959420 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:35.959388 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcab47b_bfc9_4b92_8699_24cecb8a958a.slice/crio-d795ebe7d4ff71c43f78fca7451c8b33e4bdd553ddc1103cd332808acbd3d045 WatchSource:0}: Error finding container d795ebe7d4ff71c43f78fca7451c8b33e4bdd553ddc1103cd332808acbd3d045: Status 404 returned error can't find the container with id d795ebe7d4ff71c43f78fca7451c8b33e4bdd553ddc1103cd332808acbd3d045 Oct 30 19:25:36.038749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.038724 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:36.172380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.172352 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:36.287535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.287455 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:36.290280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.290258 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\") pod \"maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:36.332189 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.332159 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" event={"ID":"abc241c2-1681-4058-97ea-980111f33d4b","Type":"ContainerStarted","Data":"0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1"} Oct 30 19:25:36.332571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.332309 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:36.333924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.333890 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" event={"ID":"0bcab47b-bfc9-4b92-8699-24cecb8a958a","Type":"ContainerStarted","Data":"bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f"} Oct 30 19:25:36.334040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.333947 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" event={"ID":"0bcab47b-bfc9-4b92-8699-24cecb8a958a","Type":"ContainerStarted","Data":"d795ebe7d4ff71c43f78fca7451c8b33e4bdd553ddc1103cd332808acbd3d045"} Oct 30 19:25:36.334040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.333977 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:36.335372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.335351 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" event={"ID":"c92245d2-8752-4350-9266-dac03282e97c","Type":"ContainerStarted","Data":"1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146"} Oct 30 19:25:36.335484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.335391 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" event={"ID":"c92245d2-8752-4350-9266-dac03282e97c","Type":"ContainerStarted","Data":"82d8babdd38ed54d654dda917a3cefcd604b6010e4e16f2224514d0f3c9b6c12"} Oct 30 19:25:36.335484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.335461 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:36.336909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.336886 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" event={"ID":"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9","Type":"ContainerStarted","Data":"778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82"} Oct 30 19:25:36.337005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.336927 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" event={"ID":"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9","Type":"ContainerStarted","Data":"40690d054d9ba1123a9f7ff3bae6dd83a62623efd156395f7857e29d3666c4f8"} Oct 30 19:25:36.337063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.337045 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:36.359925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.359857 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" podStartSLOduration=1.9787982290000001 podStartE2EDuration="2.359843552s" totalImagesPullingTime="381.045323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:36.359397976 +0000 UTC m=+506.317217166" watchObservedRunningTime="2026-10-30 19:25:36.359843552 +0000 UTC m=+506.317662741" Oct 30 19:25:36.392567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.390897 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" podStartSLOduration=2.390850997 podStartE2EDuration="2.390850997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:36.390147162 +0000 UTC m=+506.347966351" watchObservedRunningTime="2026-10-30 19:25:36.390850997 +0000 UTC m=+506.348670186" Oct 30 19:25:36.414044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.413996 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" podStartSLOduration=1.4139811199999999 podStartE2EDuration="1.41398112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:36.413680144 +0000 UTC m=+506.371499332" watchObservedRunningTime="2026-10-30 19:25:36.41398112 +0000 UTC m=+506.371800308" Oct 30 19:25:36.433661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.433610 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" podStartSLOduration=2.433594568 podStartE2EDuration="2.433594568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:36.431438373 +0000 UTC m=+506.389257563" watchObservedRunningTime="2026-10-30 19:25:36.433594568 +0000 UTC m=+506.391413757" Oct 30 19:25:36.514946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.514921 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:36.590388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.590359 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:36.787452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:36.787428 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj"] Oct 30 19:25:36.789368 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:36.789331 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8930e474_eb64_4d26_abe4_45d6f1d9c0df.slice/crio-5a545a263722c8c5ede2951e10e62ca0fccd47aecc63ce543c0facedd38979c8 WatchSource:0}: Error finding container 5a545a263722c8c5ede2951e10e62ca0fccd47aecc63ce543c0facedd38979c8: Status 404 returned error can't find the container with id 5a545a263722c8c5ede2951e10e62ca0fccd47aecc63ce543c0facedd38979c8 Oct 30 19:25:37.244594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:37.244565 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:37.344722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:37.344683 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" event={"ID":"8930e474-eb64-4d26-abe4-45d6f1d9c0df","Type":"ContainerStarted","Data":"67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b"} Oct 30 19:25:37.345179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:37.344745 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" event={"ID":"8930e474-eb64-4d26-abe4-45d6f1d9c0df","Type":"ContainerStarted","Data":"5a545a263722c8c5ede2951e10e62ca0fccd47aecc63ce543c0facedd38979c8"} Oct 30 19:25:37.346844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:37.346823 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:37.364713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:37.364672 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" podStartSLOduration=1.9522207520000001 podStartE2EDuration="2.364660571s" totalImagesPullingTime="412.439819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:37.362924941 +0000 UTC m=+507.320744129" watchObservedRunningTime="2026-10-30 19:25:37.364660571 +0000 UTC m=+507.322479758" Oct 30 19:25:37.468054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:37.467984 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.057429 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.057394 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.182907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.182853 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.300327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.300301 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.350027 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.349959 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:38.884354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.884313 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd"] Oct 30 19:25:38.889265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.889240 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:38.899221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.898989 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ad96a4-serving-cert\"" Oct 30 19:25:38.903378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.903348 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd"] Oct 30 19:25:38.959358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:38.959326 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:39.017525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.017493 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxphd\" (UniqueName: \"kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.017666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.017531 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.017707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.017660 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.017754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.017705 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.071338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.071309 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:39.118785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.118762 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.118909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.118804 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.118909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.118846 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxphd\" (UniqueName: \"kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.118909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.118887 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.119587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.119556 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.121266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.121237 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.121435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.121418 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.127849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.127827 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxphd\" (UniqueName: \"kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd\") pod \"maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.182378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.182354 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:39.208844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.208819 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:39.490112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.490088 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd"] Oct 30 19:25:39.493003 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:25:39.492964 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6153b05_0bc3_410d_885d_30da599e0594.slice/crio-c2280b589d73977140201c35fa815b5231d10fba0e85bf1f1b1c8038dba11e15 WatchSource:0}: Error finding container c2280b589d73977140201c35fa815b5231d10fba0e85bf1f1b1c8038dba11e15: Status 404 returned error can't find the container with id c2280b589d73977140201c35fa815b5231d10fba0e85bf1f1b1c8038dba11e15 Oct 30 19:25:39.541122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.541096 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:25:39.781589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.781509 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:39.901413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:39.901390 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:40.331083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:40.331055 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:25:40.366763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:40.366724 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" event={"ID":"c6153b05-0bc3-410d-885d-30da599e0594","Type":"ContainerStarted","Data":"81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8"} Oct 30 19:25:40.371077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:40.371046 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" event={"ID":"c6153b05-0bc3-410d-885d-30da599e0594","Type":"ContainerStarted","Data":"c2280b589d73977140201c35fa815b5231d10fba0e85bf1f1b1c8038dba11e15"} Oct 30 19:25:40.371295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:40.371091 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:40.392079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:40.392025 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" podStartSLOduration=2.030592507 podStartE2EDuration="2.392006557s" totalImagesPullingTime="361.41405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:40.391443239 +0000 UTC m=+510.349262429" watchObservedRunningTime="2026-10-30 19:25:40.392006557 +0000 UTC m=+510.349825747" Oct 30 19:25:40.644846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:40.644771 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:41.364069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:41.364032 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:42.164936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:42.164905 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:42.351051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:42.351025 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:25:43.104183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:43.104149 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:43.305954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:43.305924 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:25:43.306366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:43.305988 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:25:43.912486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:43.912458 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:44.362094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:44.362066 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:25:44.827143 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:44.827109 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:45.329172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:45.329147 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:25:45.329325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:45.329219 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:25:45.329394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:45.329349 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:25:45.349790 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:45.349766 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:25:45.373875 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:45.373841 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:46.253708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.253673 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:46.339297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.339268 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:25:46.339668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.339649 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:25:46.339749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.339734 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:25:46.376907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.376889 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:25:46.516232 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.516140 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:46.781298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.781229 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:46.967838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:46.967810 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:47.347665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:47.347625 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:25:47.347898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:47.347856 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:25:47.347960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:47.347941 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:25:48.040986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:48.040904 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:48.955440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:48.955410 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:49.089609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:49.089576 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:49.760968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:49.760932 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:50.160890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:50.160796 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:50.665228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:50.665196 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:50.888814 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:25:50.888774 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:25:50.913957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:50.913926 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:51.172127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:51.172100 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:51.307098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:51.307069 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:51.864149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:51.864119 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:52.042695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:52.042666 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:52.216191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:52.216156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:52.940918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:52.940851 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:53.914145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:53.914113 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:55.034136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:55.034101 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:55.417149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:55.417073 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:55.702072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:55.702039 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:56.078126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:56.078049 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:56.280164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:56.280128 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:57.662053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:57.662019 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:58.152080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:58.152045 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:58.499147 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:58.499108 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:59.131482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.131447 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:25:59.467155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.467123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:59.617667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.617639 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d"] Oct 30 19:25:59.621683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.621663 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.628629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.628608 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"openshift-service-ca.crt\"" Oct 30 19:25:59.628629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.628617 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"kube-root-ca.crt\"" Oct 30 19:25:59.636059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.636039 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"default-dockercfg-qkj8m\"" Oct 30 19:25:59.636156 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.636077 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"simulator-w4-ad96a4-kserve-self-signed-certs\"" Oct 30 19:25:59.639846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.639820 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d"] Oct 30 19:25:59.720428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.720372 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:59.723616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.723589 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j549v\" (UniqueName: \"kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.723725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.723645 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.723792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.723726 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.723792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.723761 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.723944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.723925 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.724006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.723980 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.824626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.824597 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.824772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.824645 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j549v\" (UniqueName: \"kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.824772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.824708 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.824772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.824743 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.824964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.824881 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.825017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.825002 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.825112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.825082 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.825257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.825235 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.825324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.825279 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.827104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.827074 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.827234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.827214 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.837976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.837953 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j549v\" (UniqueName: \"kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v\") pod \"simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:25:59.934707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:25:59.934678 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:26:00.212977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:00.212950 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d"] Oct 30 19:26:00.214568 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:26:00.214541 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6be95ad_7c3d_4ab6_a4d0_47f523233a1d.slice/crio-f3b0085996545f348834e3ba4a6799b4e8421c3095cf11a312cdab5cb458a543 WatchSource:0}: Error finding container f3b0085996545f348834e3ba4a6799b4e8421c3095cf11a312cdab5cb458a543: Status 404 returned error can't find the container with id f3b0085996545f348834e3ba4a6799b4e8421c3095cf11a312cdab5cb458a543 Oct 30 19:26:00.366567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:00.366535 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.470606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:00.470575 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" event={"ID":"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d","Type":"ContainerStarted","Data":"03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b"} Oct 30 19:26:00.470749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:00.470614 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" event={"ID":"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d","Type":"ContainerStarted","Data":"f3b0085996545f348834e3ba4a6799b4e8421c3095cf11a312cdab5cb458a543"} Oct 30 19:26:00.495890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:00.495790 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podStartSLOduration=1.49577918 podStartE2EDuration="1.49577918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:00.494065085 +0000 UTC m=+530.451884274" watchObservedRunningTime="2026-10-30 19:26:00.49577918 +0000 UTC m=+530.453598368" Oct 30 19:26:00.935267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:00.935242 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:26:01.226224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:01.226147 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:02.043288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:02.043254 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:02.641033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:02.641000 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:02.855166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:02.855128 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:03.593052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.593013 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:03.955745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.955714 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb"] Oct 30 19:26:03.959844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.959825 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:03.968970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.968946 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"openshift-service-ca.crt\"" Oct 30 19:26:03.969125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.968954 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"default-dockercfg-w4x7t\"" Oct 30 19:26:03.970178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.970155 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"simulator-w5-60577a-kserve-self-signed-certs\"" Oct 30 19:26:03.970311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.970241 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"kube-root-ca.crt\"" Oct 30 19:26:03.970853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:03.970836 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb"] Oct 30 19:26:04.033048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.033028 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.067665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.067632 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.067774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.067686 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.067839 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.067775 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.067940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.067921 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.068043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.068028 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.068077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.068068 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.169746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.169714 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.169928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.169797 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.169928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.169835 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.170052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.169961 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.170052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.170018 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.170161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.170100 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.170216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.170156 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.170280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.170244 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.170466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.170433 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.172551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.172524 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.173158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.173141 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.178523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.178503 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95\") pod \"simulator-w5-60577a-kserve-7cfd9755b6-555fb\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.275470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.275409 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:04.344766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.343302 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.551191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.551165 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb"] Oct 30 19:26:04.552991 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:26:04.552966 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46c3202a_06f7_4864_8039_376c19c089be.slice/crio-7681dc3720d35d118b31e46bce3d029cb712579537181388cef17f3377448f4f WatchSource:0}: Error finding container 7681dc3720d35d118b31e46bce3d029cb712579537181388cef17f3377448f4f: Status 404 returned error can't find the container with id 7681dc3720d35d118b31e46bce3d029cb712579537181388cef17f3377448f4f Oct 30 19:26:04.685901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.685854 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc"] Oct 30 19:26:04.689977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.689950 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.696163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.696139 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"kube-root-ca.crt\"" Oct 30 19:26:04.696337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.696191 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"openshift-service-ca.crt\"" Oct 30 19:26:04.696337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.696215 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"default-dockercfg-cvp5n\"" Oct 30 19:26:04.696337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.696248 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"simulator-w1-419d0a-kserve-self-signed-certs\"" Oct 30 19:26:04.703644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.703622 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc"] Oct 30 19:26:04.754575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.754549 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.774947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.774926 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.775076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.774964 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfvb\" (UniqueName: \"kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.775126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.775108 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.775172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.775156 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.775221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.775208 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.775263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.775251 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876113 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876203 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876232 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876290 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876342 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876365 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfvb\" (UniqueName: \"kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876772 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.876991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.876969 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.877154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.877133 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.879036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.878977 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.879246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.879228 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:04.886225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:04.886206 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfvb\" (UniqueName: \"kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb\") pod \"simulator-w1-419d0a-kserve-7ff5877979-ts7nc\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:05.001624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.001598 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:05.162156 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.162091 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:05.276471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.276449 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc"] Oct 30 19:26:05.278646 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:26:05.278623 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9acb067e_d374_4017_95c6_761dcde158cb.slice/crio-27a724a04e5440b9b9382b5f0486235c37072a0e92254ae39c50ac89e02ba236 WatchSource:0}: Error finding container 27a724a04e5440b9b9382b5f0486235c37072a0e92254ae39c50ac89e02ba236: Status 404 returned error can't find the container with id 27a724a04e5440b9b9382b5f0486235c37072a0e92254ae39c50ac89e02ba236 Oct 30 19:26:05.498371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.498336 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" event={"ID":"46c3202a-06f7-4864-8039-376c19c089be","Type":"ContainerStarted","Data":"cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2"} Oct 30 19:26:05.498526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.498379 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" event={"ID":"46c3202a-06f7-4864-8039-376c19c089be","Type":"ContainerStarted","Data":"7681dc3720d35d118b31e46bce3d029cb712579537181388cef17f3377448f4f"} Oct 30 19:26:05.500062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.500038 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" event={"ID":"9acb067e-d374-4017-95c6-761dcde158cb","Type":"ContainerStarted","Data":"fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501"} Oct 30 19:26:05.500159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.500076 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" event={"ID":"9acb067e-d374-4017-95c6-761dcde158cb","Type":"ContainerStarted","Data":"27a724a04e5440b9b9382b5f0486235c37072a0e92254ae39c50ac89e02ba236"} Oct 30 19:26:05.520023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.519983 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podStartSLOduration=2.519971639 podStartE2EDuration="2.519971639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:05.519550886 +0000 UTC m=+535.477370074" watchObservedRunningTime="2026-10-30 19:26:05.519971639 +0000 UTC m=+535.477790827" Oct 30 19:26:05.540078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:05.540037 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podStartSLOduration=1.54002339 podStartE2EDuration="1.54002339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:05.538098005 +0000 UTC m=+535.495917193" watchObservedRunningTime="2026-10-30 19:26:05.54002339 +0000 UTC m=+535.497842579" Oct 30 19:26:06.001719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:06.001686 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:06.040207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:06.040169 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:06.275606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:06.275537 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:06.764079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:06.764047 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:07.644250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:07.644217 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:08.657010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:08.656981 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.658523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:09.658495 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.935448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:09.935419 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:26:09.947716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:09.947692 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:26:09.951343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:09.951319 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:10.346033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:10.345971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:10.542454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:10.542429 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:26:11.136095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:11.136055 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:11.926261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:11.926233 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:12.422643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:12.422612 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:12.850585 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:12.850543 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:13.631211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:13.631181 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:14.275896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:14.275848 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:14.288312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:14.288285 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:14.341562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:14.341537 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:14.559362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:14.559292 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:26:15.002361 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:15.002328 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:15.014686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:15.014659 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:15.134333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:15.134299 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:15.564885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:15.564838 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:26:15.853096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:15.853017 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:16.546045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:16.546012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:17.372852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:17.372820 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:26:17.563025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:17.562971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:17.817339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:17.817302 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:18.648756 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:18.648723 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:19.358032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:19.358001 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:20.048537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:20.048489 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:20.740127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:20.740093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:20.881095 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:26:20.881052 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:26:21.543451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:21.543420 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:22.215109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:22.214983 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:22.924818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:22.924780 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:24.068665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.068632 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:24.398003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.397927 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:24.786991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.786962 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:24.960642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.960608 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l"] Oct 30 19:26:24.965053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.965030 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:24.974615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.974589 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"kube-root-ca.crt\"" Oct 30 19:26:24.974615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.974596 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"openshift-service-ca.crt\"" Oct 30 19:26:24.975880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.975848 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"default-dockercfg-tdws5\"" Oct 30 19:26:24.976002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.975983 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"premium-w6-01b24e-kserve-self-signed-certs\"" Oct 30 19:26:24.976534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:24.976511 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l"] Oct 30 19:26:25.020507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.020484 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:25.082426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.082338 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.082807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.082429 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.082807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.082509 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbgz\" (UniqueName: \"kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.082807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.082568 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.082807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.082617 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.082807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.082698 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184158 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184225 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184281 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184313 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbgz\" (UniqueName: \"kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184352 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184389 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184580 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184674 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.184818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.184799 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.186434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.186412 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.186813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.186790 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.193070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.193050 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbgz\" (UniqueName: \"kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz\") pod \"premium-w6-01b24e-kserve-6d5db8b568-j4x4l\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.277392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.277368 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:25.557836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.557807 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l"] Oct 30 19:26:25.559185 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:26:25.559155 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod222cea90_058a_4c0d_96a6_4e870c0b72db.slice/crio-9ea46f80633e90efbbc141dbb6a8b5c86eb5854ecd68a644b127d5b72c02deff WatchSource:0}: Error finding container 9ea46f80633e90efbbc141dbb6a8b5c86eb5854ecd68a644b127d5b72c02deff: Status 404 returned error can't find the container with id 9ea46f80633e90efbbc141dbb6a8b5c86eb5854ecd68a644b127d5b72c02deff Oct 30 19:26:25.618122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.618093 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" event={"ID":"222cea90-058a-4c0d-96a6-4e870c0b72db","Type":"ContainerStarted","Data":"9ea46f80633e90efbbc141dbb6a8b5c86eb5854ecd68a644b127d5b72c02deff"} Oct 30 19:26:25.647428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:25.647400 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:26.450811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:26.450782 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:26.624681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:26.624648 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" event={"ID":"222cea90-058a-4c0d-96a6-4e870c0b72db","Type":"ContainerStarted","Data":"f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7"} Oct 30 19:26:26.648875 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:26.648824 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podStartSLOduration=2.648810857 podStartE2EDuration="2.648810857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:26.647278177 +0000 UTC m=+556.605097365" watchObservedRunningTime="2026-10-30 19:26:26.648810857 +0000 UTC m=+556.606630044" Oct 30 19:26:27.152713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:27.152679 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:27.277496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:27.277461 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:27.834566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:27.834533 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:28.489398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:28.489368 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:28.757591 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:28.757519 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:26:28.769987 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:28.769962 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:29.528008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:29.527971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:30.361983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:30.361952 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:31.056413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:31.056383 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:31.753099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:31.753065 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:32.525189 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:32.525154 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:32.643979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:32.643950 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:33.030916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.030883 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp"] Oct 30 19:26:33.035242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.035222 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.045245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.045225 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp"] Oct 30 19:26:33.093150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.093111 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:33.125338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.125317 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"premium-w4-ad96a4-kserve-self-signed-certs\"" Oct 30 19:26:33.162657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.162630 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.162771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.162747 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.162828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.162795 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrnl\" (UniqueName: \"kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.162899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.162857 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.162899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.162894 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.162988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.162909 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.263759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.263728 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.263914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.263798 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrnl\" (UniqueName: \"kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.263914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.263851 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.263914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.263896 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.264088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.263919 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.264088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.263986 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.264182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.264162 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.264312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.264289 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.264421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.264358 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.265933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.265913 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.266276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.266257 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.272151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.272130 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrnl\" (UniqueName: \"kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl\") pod \"premium-w4-ad96a4-kserve-56f77ddb75-p2pfp\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.348636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.348581 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:33.457832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.457804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:33.628467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.628438 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp"] Oct 30 19:26:33.630137 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:26:33.630114 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cfccb13_0b2b_4acd_85ad_6fb2255cb1f3.slice/crio-9fc1aba02a2d915d9d06079b6383c0509f98a29abcd70f551ad7775ac544fac2 WatchSource:0}: Error finding container 9fc1aba02a2d915d9d06079b6383c0509f98a29abcd70f551ad7775ac544fac2: Status 404 returned error can't find the container with id 9fc1aba02a2d915d9d06079b6383c0509f98a29abcd70f551ad7775ac544fac2 Oct 30 19:26:33.662249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:33.662224 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" event={"ID":"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3","Type":"ContainerStarted","Data":"9fc1aba02a2d915d9d06079b6383c0509f98a29abcd70f551ad7775ac544fac2"} Oct 30 19:26:34.252917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:34.252853 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:34.669091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:34.669018 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" event={"ID":"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3","Type":"ContainerStarted","Data":"e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780"} Oct 30 19:26:34.688520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:34.688473 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podStartSLOduration=1.6884609400000001 podStartE2EDuration="1.68846094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:34.687993095 +0000 UTC m=+564.645812283" watchObservedRunningTime="2026-10-30 19:26:34.68846094 +0000 UTC m=+564.646280127" Oct 30 19:26:35.057105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.057074 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:35.278256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.278226 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:35.290751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.290729 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:35.349700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.349643 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:35.411786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.411760 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt"] Oct 30 19:26:35.421912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.421882 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.427503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.427472 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt"] Oct 30 19:26:35.427722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.427705 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"premium-w1-419d0a-kserve-self-signed-certs\"" Oct 30 19:26:35.475073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.475050 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:35.587244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.587211 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.587418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.587270 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.587418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.587372 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dhf\" (UniqueName: \"kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.587418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.587411 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.587584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.587488 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.587644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.587628 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.685609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.685584 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:26:35.688110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688090 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688126 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688158 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dhf\" (UniqueName: \"kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688266 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688310 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688372 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688448 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688530 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.688603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.688580 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.690447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.690420 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.690746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.690730 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.696786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.696767 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dhf\" (UniqueName: \"kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf\") pod \"premium-w1-419d0a-kserve-6dbd9d446b-5spqt\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.736791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.736764 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:35.837135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:35.837103 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:36.018165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:36.018142 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt"] Oct 30 19:26:36.019817 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:26:36.019790 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7d8af65_c587_4d75_8161_b40d13e320a1.slice/crio-56d8cc13a413e81ea0d8c080458c1b448039d06f93cba07b85d88f1ba5e32485 WatchSource:0}: Error finding container 56d8cc13a413e81ea0d8c080458c1b448039d06f93cba07b85d88f1ba5e32485: Status 404 returned error can't find the container with id 56d8cc13a413e81ea0d8c080458c1b448039d06f93cba07b85d88f1ba5e32485 Oct 30 19:26:36.662413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:36.662372 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:36.691432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:36.691397 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" event={"ID":"a7d8af65-c587-4d75-8161-b40d13e320a1","Type":"ContainerStarted","Data":"c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda"} Oct 30 19:26:36.691918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:36.691439 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" event={"ID":"a7d8af65-c587-4d75-8161-b40d13e320a1","Type":"ContainerStarted","Data":"56d8cc13a413e81ea0d8c080458c1b448039d06f93cba07b85d88f1ba5e32485"} Oct 30 19:26:36.713668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:36.713621 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podStartSLOduration=1.713607836 podStartE2EDuration="1.713607836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:36.712450314 +0000 UTC m=+566.670269503" watchObservedRunningTime="2026-10-30 19:26:36.713607836 +0000 UTC m=+566.671427023" Oct 30 19:26:36.737231 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:36.737204 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:37.545232 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:37.545201 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:38.418365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:38.418337 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:39.360716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:39.360647 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:40.345026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:40.344995 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:41.219368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:41.219322 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:41.948757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:41.948727 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:42.656978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:42.656947 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:43.348769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:43.348730 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:43.361052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:43.361029 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:43.718306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:43.718271 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:43.741201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:43.741179 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:26:44.453237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:44.453207 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:45.532525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:45.532492 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:45.737213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:45.737189 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:45.750423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:45.750402 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:46.352877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:46.352835 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:46.749133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:46.749102 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:26:47.135301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:47.135226 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:47.948010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:47.947978 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:48.644882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:48.644840 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:49.336695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:49.336667 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:50.055643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:50.055612 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:50.591799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:50.591769 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:26:50.881344 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:26:50.881262 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:26:50.923903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:50.923842 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.042842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:52.042807 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.292804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:52.292777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.406805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:52.406737 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.838906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:52.838852 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:53.647718 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:53.647684 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:54.626859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:54.626826 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:55.449518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:55.449483 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:56.204236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:56.204185 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.098272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:57.098239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.993982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:57.993948 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:58.764174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:58.764140 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:59.439325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:26:59.439294 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:00.295436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:00.295405 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:00.955232 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:00.955198 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:01.640781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:01.640744 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:02.341300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:02.341268 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:03.044994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:03.044962 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:03.744203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:03.744168 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:04.542884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:04.542831 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:05.239384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:05.239351 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:05.934850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:05.934818 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:06.642096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:06.642061 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:07.451349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:07.451317 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:07.567886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:07.567838 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:08.359795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:08.359754 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:09.357180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:09.357139 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:09.614943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:09.614851 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:10.000810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:10.000781 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:10.222383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:10.222350 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:27:10.222564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:10.222414 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:27:10.466546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:10.466516 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:11.365216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:11.365183 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:12.139159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:12.139120 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:12.830924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:12.830888 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:13.647165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:13.647132 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:14.441857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:14.441822 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:15.258496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:15.258448 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:15.938904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:15.938848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:16.643279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:16.643245 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:17.347151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:17.347119 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:17.531982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:17.531951 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:27:18.148593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:18.148563 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:18.847903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:18.847850 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:19.642935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:19.642876 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:20.322187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:20.322150 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:20.881078 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:27:20.881041 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:27:21.136124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:21.136049 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:22.185871 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:22.185826 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:22.925985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:22.925952 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:23.658738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:23.658706 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:24.337468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:24.337433 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:25.270268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:25.270237 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:26.046906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.046851 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:26.353974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.353896 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv"] Oct 30 19:27:26.358390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.358363 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.370026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.370001 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv"] Oct 30 19:27:26.398045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.398020 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.398174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.398135 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.398174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.398170 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvd9p\" (UniqueName: \"kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.427389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.427363 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:26.427484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.427440 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:27:26.498991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.498960 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.499155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.498999 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvd9p\" (UniqueName: \"kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.499155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.499097 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.499645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.499620 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.501409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.501391 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.507775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.507753 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvd9p\" (UniqueName: \"kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p\") pod \"maas-api-revoke-sub-23af4310-qvjpv\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.673480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.673417 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:26.877055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.877023 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:26.948087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.948059 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv"] Oct 30 19:27:26.949776 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:27:26.949747 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c05307_98d1_4ca4_8b84_28ca13dd3f3d.slice/crio-c3887dcf634a20a31099c996e347823be58dade21329184219dbc7e829e35215 WatchSource:0}: Error finding container c3887dcf634a20a31099c996e347823be58dade21329184219dbc7e829e35215: Status 404 returned error can't find the container with id c3887dcf634a20a31099c996e347823be58dade21329184219dbc7e829e35215 Oct 30 19:27:26.994723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:26.994696 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" event={"ID":"91c05307-98d1-4ca4-8b84-28ca13dd3f3d","Type":"ContainerStarted","Data":"c3887dcf634a20a31099c996e347823be58dade21329184219dbc7e829e35215"} Oct 30 19:27:27.656445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:27.656412 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:28.357726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:28.357690 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:29.153882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:29.153840 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:29.853692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:29.853668 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:30.013927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:30.013891 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" event={"ID":"91c05307-98d1-4ca4-8b84-28ca13dd3f3d","Type":"ContainerStarted","Data":"aeabf6c0fe08b88c5284247dfed932a3850d0e8f9d84d7a5bd4cf56aa2daf744"} Oct 30 19:27:30.034360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:30.034311 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" podStartSLOduration=1.4332315310000001 podStartE2EDuration="4.034298491s" totalImagesPullingTime="2.60106696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:30.032589396 +0000 UTC m=+619.990408582" watchObservedRunningTime="2026-10-30 19:27:30.034298491 +0000 UTC m=+619.992117677" Oct 30 19:27:30.650074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:30.649993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:31.020801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:31.020773 6514 generic.go:356] "Generic (PLEG): container finished" podID="91c05307-98d1-4ca4-8b84-28ca13dd3f3d" containerID="aeabf6c0fe08b88c5284247dfed932a3850d0e8f9d84d7a5bd4cf56aa2daf744" exitCode=0 Oct 30 19:27:31.020962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:31.020888 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" event={"ID":"91c05307-98d1-4ca4-8b84-28ca13dd3f3d","Type":"ContainerDied","Data":"aeabf6c0fe08b88c5284247dfed932a3850d0e8f9d84d7a5bd4cf56aa2daf744"} Oct 30 19:27:31.405347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:31.405260 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:32.157151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.157127 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:32.263327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.263297 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p\" (UniqueName: \"kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p\") pod \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " Oct 30 19:27:32.263511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.263379 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca\") pod \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " Oct 30 19:27:32.263511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.263438 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca\") pod \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\" (UID: \"91c05307-98d1-4ca4-8b84-28ca13dd3f3d\") " Oct 30 19:27:32.263855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.263826 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca" pod "91c05307-98d1-4ca4-8b84-28ca13dd3f3d" (UID: "91c05307-98d1-4ca4-8b84-28ca13dd3f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:32.265462 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.265426 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p" pod "91c05307-98d1-4ca4-8b84-28ca13dd3f3d" (UID: "91c05307-98d1-4ca4-8b84-28ca13dd3f3d"). InnerVolumeSpecName "kube-api-access-xvd9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:32.265590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.265534 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca" pod "91c05307-98d1-4ca4-8b84-28ca13dd3f3d" (UID: "91c05307-98d1-4ca4-8b84-28ca13dd3f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:32.279159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.279105 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:32.364583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.364547 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvd9p\" (UniqueName: \"kubernetes.io/projected/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-kube-api-access-xvd9p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.364708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.364589 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.364708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:32.364607 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c05307-98d1-4ca4-8b84-28ca13dd3f3d-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:33.033188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:33.033159 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" Oct 30 19:27:33.033347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:33.033163 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv" event={"ID":"91c05307-98d1-4ca4-8b84-28ca13dd3f3d","Type":"ContainerDied","Data":"c3887dcf634a20a31099c996e347823be58dade21329184219dbc7e829e35215"} Oct 30 19:27:33.033347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:33.033279 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3887dcf634a20a31099c996e347823be58dade21329184219dbc7e829e35215" Oct 30 19:27:33.115935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:33.115908 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:33.947138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:33.947106 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:34.799968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:34.799934 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:35.616725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:35.616690 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:36.451461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:36.451412 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:37.200163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:37.200131 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:38.049939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:38.049903 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:38.862674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:38.862640 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:39.678509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:39.678476 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:40.354236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:40.354196 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:41.156330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:41.156299 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:41.828087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:41.828038 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:42.532601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:42.532568 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:43.210023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:43.209985 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:43.843036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:43.842989 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:44.042819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:44.042781 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:27:44.551155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:44.551124 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:45.416489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:45.416450 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:46.337235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:46.337201 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:47.093824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:47.093791 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:48.134390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:48.134319 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:48.940601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:48.940571 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:49.350342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:49.350268 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:50.000012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:49.999983 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:50.749569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:50.749538 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:50.880378 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:27:50.880334 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:27:51.555341 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:51.555291 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:52.345978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:52.345882 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:53.150930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:53.150884 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:53.955190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:53.955156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:54.671991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:54.671953 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:55.489743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:55.489641 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:56.282163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.282125 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9"] Oct 30 19:27:56.282745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.282721 6514 cpu_manager.go:414] "containerMap: removing container" podUID="91c05307-98d1-4ca4-8b84-28ca13dd3f3d" containerName="revoke-keys" Oct 30 19:27:56.282745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.282741 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c05307-98d1-4ca4-8b84-28ca13dd3f3d" containerName="revoke-keys" Oct 30 19:27:56.282930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.282890 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c05307-98d1-4ca4-8b84-28ca13dd3f3d" containerName="revoke-keys" Oct 30 19:27:56.286934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.286914 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.291885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.291853 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-01b24e-openshift-default-dockercfg-tz4v8\"" Oct 30 19:27:56.300084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.300062 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9"] Oct 30 19:27:56.302032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.302009 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:56.415338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415293 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5bd\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415396 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415466 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415503 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415522 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415695 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415750 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415810 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.415858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.415843 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.483524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.483499 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:56.521119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.521078 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.521682 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.521650 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.521919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.521895 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522079 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522232 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522405 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522420 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522469 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522583 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5bd\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522632 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522652 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522714 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.522921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.522902 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.523402 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.523372 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.524926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.524903 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.525952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.525920 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.540363 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.540209 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.540814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.540780 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5bd\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.551487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.551464 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw"] Oct 30 19:27:56.557225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.557208 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.562087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.562065 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-60577a-openshift-default-dockercfg-86d5l\"" Oct 30 19:27:56.569155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.569128 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw"] Oct 30 19:27:56.604106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.604082 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:27:56.699913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.698954 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9"] Oct 30 19:27:56.704826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.704802 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.710558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.710537 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-419d0a-openshift-default-dockercfg-tghgq\"" Oct 30 19:27:56.722468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.722430 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9"] Oct 30 19:27:56.724657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724639 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.724737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724674 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.724737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724701 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.724852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724838 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.724919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724883 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.724919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724906 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24pg\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.725007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724925 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.725007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.724954 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.725110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.725061 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.825636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825572 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.825636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825616 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.825813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825685 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.825813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825709 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g24pg\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.825813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825737 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjz5r\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825812 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825946 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.825982 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826026 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826133 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826154 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826178 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826189 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826272 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826305 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826314 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826383 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826452 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826506 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826578 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.826654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826586 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.826654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.826634 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.827218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.827196 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.828350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.828333 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.828484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.828468 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.837153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.837132 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.837228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.837169 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24pg\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.880212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.880184 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:27:56.890687 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.890658 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9"] Oct 30 19:27:56.899899 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:27:56.899855 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4b2279_7bef_4b97_ab14_28417776470e.slice/crio-8e4e71560b67a5137aa634a048178f553c6ca26b223f6d275757b4df8ae04110 WatchSource:0}: Error finding container 8e4e71560b67a5137aa634a048178f553c6ca26b223f6d275757b4df8ae04110: Status 404 returned error can't find the container with id 8e4e71560b67a5137aa634a048178f553c6ca26b223f6d275757b4df8ae04110 Oct 30 19:27:56.928121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928093 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928168 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928212 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928275 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjz5r\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928312 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928418 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928526 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928544 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928575 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.928625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.928615 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.929933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.929596 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.929933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.929602 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.929933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.929825 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.929933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.929901 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.931466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.931437 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.932832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.932804 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.939743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.939719 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjz5r\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:56.941016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:56.940969 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:57.022061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.022035 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:27:57.042563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.042526 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2"] Oct 30 19:27:57.049729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.049706 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.061728 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.060988 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad96a4-openshift-default-dockercfg-6zjhk\"" Oct 30 19:27:57.063295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.063273 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2"] Oct 30 19:27:57.179770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.179748 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw"] Oct 30 19:27:57.181496 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:27:57.181460 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90471c40_e5e4_400d_9838_18151f77cd3c.slice/crio-a09e1661e020ebf21ca552d0906fad9f78dd2647e177f446151716e4bc91bfc6 WatchSource:0}: Error finding container a09e1661e020ebf21ca552d0906fad9f78dd2647e177f446151716e4bc91bfc6: Status 404 returned error can't find the container with id a09e1661e020ebf21ca552d0906fad9f78dd2647e177f446151716e4bc91bfc6 Oct 30 19:27:57.186360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.186329 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" event={"ID":"bb4b2279-7bef-4b97-ab14-28417776470e","Type":"ContainerStarted","Data":"8e4e71560b67a5137aa634a048178f553c6ca26b223f6d275757b4df8ae04110"} Oct 30 19:27:57.233198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233172 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233215 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233355 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233424 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233459 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233495 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233568 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233825 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.233966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.233877 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpx85\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.334826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.334801 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.334961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.334831 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpx85\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.334961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.334902 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.334961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.334930 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.334961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.334950 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.334999 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335050 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335094 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335165 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335212 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335255 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335433 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335532 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.335677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.335652 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.337201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.337180 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.337476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.337455 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.356808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.356750 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.357334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.357310 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpx85\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.358236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.358208 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9"] Oct 30 19:27:57.363067 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:27:57.361729 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b58f0f0_0f6a_4189_92c0_0921200c5c3d.slice/crio-33715a9b3600750fe2c9ef1779ae755390483f87abe3852a92e197d8f7a629f7 WatchSource:0}: Error finding container 33715a9b3600750fe2c9ef1779ae755390483f87abe3852a92e197d8f7a629f7: Status 404 returned error can't find the container with id 33715a9b3600750fe2c9ef1779ae755390483f87abe3852a92e197d8f7a629f7 Oct 30 19:27:57.370066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.370045 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:27:57.473554 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.473521 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:57.675956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:57.675919 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2"] Oct 30 19:27:57.677804 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:27:57.677778 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501abc44_e5b5_4b47_bce4_773ccca705e7.slice/crio-eb43d781d04235cd6e264dbd07d3e4ff0f96afccce9010177e917e339e48372f WatchSource:0}: Error finding container eb43d781d04235cd6e264dbd07d3e4ff0f96afccce9010177e917e339e48372f: Status 404 returned error can't find the container with id eb43d781d04235cd6e264dbd07d3e4ff0f96afccce9010177e917e339e48372f Oct 30 19:27:58.197445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:58.197399 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" event={"ID":"90471c40-e5e4-400d-9838-18151f77cd3c","Type":"ContainerStarted","Data":"a09e1661e020ebf21ca552d0906fad9f78dd2647e177f446151716e4bc91bfc6"} Oct 30 19:27:58.199269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:58.199199 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" event={"ID":"501abc44-e5b5-4b47-bce4-773ccca705e7","Type":"ContainerStarted","Data":"eb43d781d04235cd6e264dbd07d3e4ff0f96afccce9010177e917e339e48372f"} Oct 30 19:27:58.201078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:58.201044 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" event={"ID":"5b58f0f0-0f6a-4189-92c0-0921200c5c3d","Type":"ContainerStarted","Data":"33715a9b3600750fe2c9ef1779ae755390483f87abe3852a92e197d8f7a629f7"} Oct 30 19:27:58.450256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:58.450161 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:59.386699 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:27:59.386668 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:00.351359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:00.351324 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:01.387223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:01.387194 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:01.964838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:01.964799 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:02.006717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.006618 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.006717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.006702 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.007087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.006770 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.184660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.184607 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.184933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.184909 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.185092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.185074 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.245399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.245367 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" event={"ID":"90471c40-e5e4-400d-9838-18151f77cd3c","Type":"ContainerStarted","Data":"b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4"} Oct 30 19:28:02.272288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.272243 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podStartSLOduration=1.450767428 podStartE2EDuration="6.272215264s" totalImagesPullingTime="4.821447836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:02.269546704 +0000 UTC m=+652.227365895" watchObservedRunningTime="2026-10-30 19:28:02.272215264 +0000 UTC m=+652.230034453" Oct 30 19:28:02.365687 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.365649 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.365814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.365753 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.365910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.365815 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.367115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.367081 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:02.415521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.409369 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.415521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.409471 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.415521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.409511 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:02.888942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:02.884503 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:28:03.258954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.258916 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" event={"ID":"501abc44-e5b5-4b47-bce4-773ccca705e7","Type":"ContainerStarted","Data":"3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0"} Oct 30 19:28:03.262943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.262904 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" event={"ID":"bb4b2279-7bef-4b97-ab14-28417776470e","Type":"ContainerStarted","Data":"42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503"} Oct 30 19:28:03.266524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.266495 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" event={"ID":"5b58f0f0-0f6a-4189-92c0-0921200c5c3d","Type":"ContainerStarted","Data":"0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9"} Oct 30 19:28:03.290260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.290209 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podStartSLOduration=1.562705622 podStartE2EDuration="6.290193301s" totalImagesPullingTime="4.727487679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:03.287399898 +0000 UTC m=+653.245219093" watchObservedRunningTime="2026-10-30 19:28:03.290193301 +0000 UTC m=+653.248012558" Oct 30 19:28:03.360889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.360449 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podStartSLOduration=2.360662972 podStartE2EDuration="7.36042816s" totalImagesPullingTime="4.999765188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:03.332599481 +0000 UTC m=+653.290418670" watchObservedRunningTime="2026-10-30 19:28:03.36042816 +0000 UTC m=+653.318247349" Oct 30 19:28:03.380374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.373498 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:28:03.384889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.382941 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:03.384889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.383037 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:03.591069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.577883 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:03.624010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.623962 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:28:03.663889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.638944 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:03.663889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.639034 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:03.946049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.913455 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:03.946049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:03.934283 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:04.026090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.026056 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:28:04.026477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.026390 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:04.026656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.026628 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:04.376983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.375966 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:04.376983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.376029 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:04.550888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.549820 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:04.565888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.553942 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:04.583888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.569152 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:28:04.614893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.607327 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:04.614893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.607401 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:04.781891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.781253 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:04.781891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.781659 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:28:04.806885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.786518 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:04.899887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.899259 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:04.899887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.899322 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:05.018893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:04.993107 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:05.062943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.060248 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:05.062943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.060321 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:05.284374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.283943 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 19:28:05.373378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.373266 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:05.373642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.373600 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:05.555916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.554956 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:05.555916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.555594 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:28:05.565977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.557036 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:05.565977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.557356 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:28:05.635939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.633839 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:05.635939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.633917 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:05.662354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.662293 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podStartSLOduration=4.380680466 podStartE2EDuration="9.662273295s" totalImagesPullingTime="5.281592829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:03.373213875 +0000 UTC m=+653.331033063" watchObservedRunningTime="2026-10-30 19:28:05.662273295 +0000 UTC m=+655.620092482" Oct 30 19:28:05.782001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.781952 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:05.782588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.782562 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:05.783282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.783262 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:28:05.786999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.786971 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:28:05.818912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.792322 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:05.826885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.819726 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:28:05.914943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:05.898854 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:06.015034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.014954 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:06.065010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.064958 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:28:06.151886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.145058 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.189945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.152190 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:06.189945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.152270 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:06.189945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.152551 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:28:06.288896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.276256 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:06.303892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.289430 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.741784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.408445 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:06.744996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.744943 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:06.745204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.588063 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.749364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.588105 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:06.754007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.753950 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.754180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.587981 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.759332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.723752 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.783715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.783637 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.785480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.784015 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:06.802067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.802026 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:06.802321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.802281 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:06.828149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.828106 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:07.025946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:06.976325 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:07.027252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.027214 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:07.027413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.026903 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:07.027539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.027521 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:07.167943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.165363 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:07.372943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.370433 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:07.373206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.373173 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:07.378571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.378548 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:28:07.684888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.680585 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:07.684888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.680661 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:07.786888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.783061 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:28:07.812295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.794767 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:07.863967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.794805 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:07.867914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.794825 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:07.876884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.817920 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:28:07.901488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.874949 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:28:07.973602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.972985 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:07.973602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:07.973041 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:08.024244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.024198 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:08.024492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.024461 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:08.169034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.168976 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:28:08.392235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.392040 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:08.392235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.392099 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:08.537888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.537151 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:08.607386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.607309 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:08.607669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.607596 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:08.815002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.814948 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:08.828604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.827584 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:08.868086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.868028 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:08.881256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.881209 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:08.881511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.881485 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:08.909153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.909114 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:08.928674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:08.918046 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:09.023326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.023286 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:09.023601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.023570 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:09.171942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.170950 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:09.277898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.276809 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:28:09.304274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.303106 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:28:09.414893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.410753 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:09.414893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.410809 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:09.470908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.470184 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:28:09.487341 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.487309 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:28:09.488040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.488002 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:28:09.488950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.488848 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:28:09.611889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.611707 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:28:09.611889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.611760 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:28:09.860698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.860612 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:09.900980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.900948 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:28:09.901265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:09.901233 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 19:28:10.022923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.022885 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:28:10.023141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.022945 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 19:28:10.175335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.175240 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:10.304564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.304487 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:10.370674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.370638 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:28:10.370966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.370931 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:28:10.610647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.610618 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:28:10.611268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.611240 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:28:10.612408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.612378 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:28:10.847145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.847117 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:10.903383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.903304 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:28:10.903964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.903925 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:28:10.908394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:10.908377 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:28:11.028986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.028963 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:28:11.029531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.029507 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:28:11.030468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.030447 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:28:11.376382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.376346 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:28:11.376842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.376824 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:28:11.377845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.377825 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:28:11.423054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.423011 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x"] Oct 30 19:28:11.429444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.429426 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.436189 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.436169 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b623ca-openshift-default-dockercfg-9vknm\"" Oct 30 19:28:11.447672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.447653 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x"] Oct 30 19:28:11.484245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484201 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484248 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484338 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484491 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484562 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484631 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484693 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484746 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwz2t\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.484827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.484789 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585426 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585478 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585531 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585574 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwz2t\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585615 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585654 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585684 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585717 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.585856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.585804 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.586291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.586265 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.586489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.586467 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.586603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.586533 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.586603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.586556 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.586827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.586799 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.588086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.588064 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.588613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.588597 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.601919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.601895 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwz2t\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.602373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.602353 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.742359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.742337 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:11.857135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:11.857070 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:12.034825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.034761 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x"] Oct 30 19:28:12.037159 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:12.037136 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dfe9fc_35dc_4437_8475_f31edf469aae.slice/crio-510c944da25c046c8244765ab74b03a9b46431873a0402279f2195592dda4060 WatchSource:0}: Error finding container 510c944da25c046c8244765ab74b03a9b46431873a0402279f2195592dda4060: Status 404 returned error can't find the container with id 510c944da25c046c8244765ab74b03a9b46431873a0402279f2195592dda4060 Oct 30 19:28:12.039497 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.039462 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:12.039566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.039529 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:12.039566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.039558 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:12.078480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.078333 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" event={"ID":"11dfe9fc-35dc-4437-8475-f31edf469aae","Type":"ContainerStarted","Data":"510c944da25c046c8244765ab74b03a9b46431873a0402279f2195592dda4060"} Oct 30 19:28:12.336730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.336637 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:12.650092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:12.650026 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:13.086528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:13.086491 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" event={"ID":"11dfe9fc-35dc-4437-8475-f31edf469aae","Type":"ContainerStarted","Data":"e63a5a2047f335e4a9810191d411dcf35945d8b7142764ea03c5496f3f3d1f32"} Oct 30 19:28:13.119303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:13.119241 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podStartSLOduration=2.119223288 podStartE2EDuration="2.119223288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:13.11031386 +0000 UTC m=+663.068133049" watchObservedRunningTime="2026-10-30 19:28:13.119223288 +0000 UTC m=+663.077042476" Oct 30 19:28:13.569104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:13.569078 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:13.743448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:13.743416 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:13.745204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:13.745174 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:28:13.745343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:13.745222 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:28:14.323903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.320434 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n"] Oct 30 19:28:14.325611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.325586 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n"] Oct 30 19:28:14.325802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.325722 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.339841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.339809 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.340071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.339911 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.340071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.339946 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.340071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.340017 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.341045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.340090 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.341045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.340136 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.341045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.340248 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.341045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.340308 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.341045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.340368 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9zj\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.342514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.342489 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gt4zp\"" Oct 30 19:28:14.388928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.388831 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:14.442426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442344 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442399 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442449 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9zj\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442474 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442510 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442527 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442544 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442583 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.442781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.442613 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.443230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.443199 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.443340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.443229 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.443412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.443399 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.443658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.443630 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.443855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.443836 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.445627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.445604 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.445751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.445730 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.454165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.454144 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9zj\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.454769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.454745 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4g9n\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.512883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.508353 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:14.648137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.648061 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:14.744892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.743407 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:28:14.744892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.743467 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:28:14.956434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.956409 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n"] Oct 30 19:28:14.957272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:14.957246 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839f94be_9343_4fc5_830b_d489bbdb4059.slice/crio-5eb366c6fd70e58474019f7e5556fb7ebbf73db734505de86118bc935b89b425 WatchSource:0}: Error finding container 5eb366c6fd70e58474019f7e5556fb7ebbf73db734505de86118bc935b89b425: Status 404 returned error can't find the container with id 5eb366c6fd70e58474019f7e5556fb7ebbf73db734505de86118bc935b89b425 Oct 30 19:28:14.959709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.959676 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:14.959799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.959744 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:14.959799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:14.959776 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:15.128443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.128404 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" event={"ID":"839f94be-9343-4fc5-830b-d489bbdb4059","Type":"ContainerStarted","Data":"1c3ba15c970e28e9b1b44493207109256d46d1a496784b86bf66f70401d5a2d0"} Oct 30 19:28:15.128622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.128465 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" event={"ID":"839f94be-9343-4fc5-830b-d489bbdb4059","Type":"ContainerStarted","Data":"5eb366c6fd70e58474019f7e5556fb7ebbf73db734505de86118bc935b89b425"} Oct 30 19:28:15.154962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.154685 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podStartSLOduration=1.154666349 podStartE2EDuration="1.154666349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:15.153487219 +0000 UTC m=+665.111306433" watchObservedRunningTime="2026-10-30 19:28:15.154666349 +0000 UTC m=+665.112485538" Oct 30 19:28:15.649153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.649107 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:15.725899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.724478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:15.743370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.743341 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:28:15.743586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:15.743563 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:28:16.087841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:16.087811 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:28:16.650900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:16.650872 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:16.651405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:16.650946 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:16.743565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:16.743519 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:28:16.743772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:16.743586 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:28:16.862071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:16.862040 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:17.652844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:17.652806 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:17.653461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:17.653425 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:17.743316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:17.743276 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:28:17.743595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:17.743563 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:28:18.649384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:18.649339 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:18.649564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:18.649414 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:18.706355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:18.706323 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:18.745347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:18.745315 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:28:18.745612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:18.745572 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:28:19.649377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:19.649340 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:19.649535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:19.649423 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:19.779350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:19.779312 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:20.534074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:20.533100 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:20.534074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:20.533889 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:20.534767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:20.534750 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:28:20.648509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:20.648474 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:20.648661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:20.648551 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:20.753387 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:20.753352 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:20.887327 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:28:20.887232 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:28:21.316706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:21.316672 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:21.634277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:21.634181 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:21.649137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:21.649111 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:21.649229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:21.649178 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:22.308134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:22.308100 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:22.649230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:22.649154 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:28:22.649230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:22.649201 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 19:28:22.954002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:22.953969 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:23.069040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:23.068995 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:23.844100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:23.844062 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:23.981142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:23.981114 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:23.981588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:23.981560 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:23.982543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:23.982524 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:28:24.615384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:24.615354 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:25.437023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:25.436989 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:26.234856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.234826 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:26.610960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.610884 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8"] Oct 30 19:28:26.614961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.614940 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.632602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.632576 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8"] Oct 30 19:28:26.778454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778427 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zkr\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778473 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778552 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778622 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778667 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778699 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778728 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.778930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.778903 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.779053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.779035 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880293 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880338 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880388 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880436 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880502 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zkr\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880529 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880717 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880735 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880759 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.880852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.880808 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.881229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.881196 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.881364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.881254 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.881476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.881458 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.881548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.881477 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.882971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.882943 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.883169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.883151 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.891668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.891648 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.891855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.891837 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zkr\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.936633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.936612 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:26.963359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:26.963333 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:27.164024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.163946 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz"] Oct 30 19:28:27.168767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.168745 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.177978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.177949 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz"] Oct 30 19:28:27.178513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.178494 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ae63bf-openshift-default-dockercfg-2n7kg\"" Oct 30 19:28:27.250261 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:27.250228 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e40281f_cfcb_458a_b4c9_3216c0489992.slice/crio-e97d3b40dc408d8cf94a63776b5ccfd8d09843ad752d50f138e3d24eb964b6a3 WatchSource:0}: Error finding container e97d3b40dc408d8cf94a63776b5ccfd8d09843ad752d50f138e3d24eb964b6a3: Status 404 returned error can't find the container with id e97d3b40dc408d8cf94a63776b5ccfd8d09843ad752d50f138e3d24eb964b6a3 Oct 30 19:28:27.252494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.252463 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8"] Oct 30 19:28:27.253172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.253136 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:27.253255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.253219 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:27.253310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.253267 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:27.289312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289284 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.289425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289405 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.289494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289446 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.289621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289592 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.289687 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289652 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gnl\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.289787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289761 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.289939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.289918 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.290119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.290062 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.290119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.290100 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.294926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.294899 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" event={"ID":"2e40281f-cfcb-458a-b4c9-3216c0489992","Type":"ContainerStarted","Data":"e97d3b40dc408d8cf94a63776b5ccfd8d09843ad752d50f138e3d24eb964b6a3"} Oct 30 19:28:27.390931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.390901 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391019 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391071 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391118 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391149 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gnl\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391208 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391237 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391320 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391346 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391346 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391830 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391801 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.391981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.391892 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.392055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.392033 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.392311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.392288 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.393771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.393746 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.394024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.394005 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.400287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.400267 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gnl\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.400422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.400407 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.490351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.490316 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:27.652431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.652401 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:27.775542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.775508 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz"] Oct 30 19:28:27.782291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.782257 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:27.782395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.782328 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:27.782395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:27.782356 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:28.324225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.324187 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" event={"ID":"2e40281f-cfcb-458a-b4c9-3216c0489992","Type":"ContainerStarted","Data":"4934838d90e3cf4bac17fe83d4d5ad0ad5c68d28fe5301e19477a2a1e1b8f6d0"} Oct 30 19:28:28.326023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.325992 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" event={"ID":"e814ff8d-5637-463d-9171-29c0c699182e","Type":"ContainerStarted","Data":"ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49"} Oct 30 19:28:28.326162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.326049 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" event={"ID":"e814ff8d-5637-463d-9171-29c0c699182e","Type":"ContainerStarted","Data":"2aa2a579a39c0df57d1e0fd1ae97379db75b8cf9d4da2918eae46a9b965e177d"} Oct 30 19:28:28.347993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.347945 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podStartSLOduration=2.347932843 podStartE2EDuration="2.347932843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:28.344126074 +0000 UTC m=+678.301945261" watchObservedRunningTime="2026-10-30 19:28:28.347932843 +0000 UTC m=+678.305752033" Oct 30 19:28:28.367522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.367478 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podStartSLOduration=1.3674676749999999 podStartE2EDuration="1.367467675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:28.364208787 +0000 UTC m=+678.322027979" watchObservedRunningTime="2026-10-30 19:28:28.367467675 +0000 UTC m=+678.325286863" Oct 30 19:28:28.490456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.490423 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:28.500008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.499974 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:28.500303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.500271 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:28.552717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.552669 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:28.743933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.742706 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:28.939589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.939178 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:28.939589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.939293 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:28:28.939589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:28.939338 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:28:29.491184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:29.491152 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:29.491440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:29.491411 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:29.609887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:29.602430 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:29.949888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:29.945507 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:28:29.949888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:29.945564 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:28:30.519889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:30.490888 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:30.519889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:30.490961 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:30.547261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:30.546513 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:30.944486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:30.944454 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:28:30.944826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:30.944795 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:28:31.485531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:31.485170 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:31.507888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:31.495207 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:31.507888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:31.495261 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:31.947065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:31.939965 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:28:31.947065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:31.940024 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:28:32.494288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:32.494252 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:32.494880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:32.494830 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:32.959888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:32.944101 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:28:32.959888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:32.944162 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:28:33.492287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.492251 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:33.492674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.492644 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:33.494656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.494613 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:33.762048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.761949 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:33.835887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.834973 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:33.936984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.936944 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:33.943896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.942972 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:28:33.943896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:33.943043 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 19:28:34.209887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:34.186324 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:34.501254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:34.494625 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:28:34.503967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:34.503913 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 19:28:34.859506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:34.859429 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:35.770676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:35.770630 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:35.986044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:35.985999 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:35.986238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:35.986083 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:36.022951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.019623 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:36.022951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.021008 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:36.042924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.042579 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:36.108186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.108153 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:28:36.321524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.321450 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:36.322030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.322009 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:36.323303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.323281 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:28:36.665092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:36.665017 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:37.462029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:37.461972 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:38.269572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:38.269543 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:38.976161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:38.976129 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:39.682184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:39.681778 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:40.551348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:40.551308 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:41.246916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.246883 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:41.366643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.366598 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q"] Oct 30 19:28:41.375526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.375503 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.389799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.389776 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q"] Oct 30 19:28:41.458860 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.458826 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.458860 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.458874 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.458895 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.459061 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.459203 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.459245 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.459339 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.459405 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthbd\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.459479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.459440 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560193 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthbd\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560237 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560278 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560296 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560314 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560351 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560451 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560488 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560537 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.560735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560706 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.561040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560793 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.561040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560934 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.561040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.560947 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.561203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.561181 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.562562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.562539 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.562976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.562956 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.583261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.583236 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthbd\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.589481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.589460 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.692886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.692840 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:41.976945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.976918 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q"] Oct 30 19:28:41.978169 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:41.978143 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45155a92_257c_47b0_a18e_38bff73f2120.slice/crio-90508dc5a740d4d02a8beeddf0242bd337e0adfc2f10295ab4367a58509e0a4c WatchSource:0}: Error finding container 90508dc5a740d4d02a8beeddf0242bd337e0adfc2f10295ab4367a58509e0a4c: Status 404 returned error can't find the container with id 90508dc5a740d4d02a8beeddf0242bd337e0adfc2f10295ab4367a58509e0a4c Oct 30 19:28:41.980447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.980405 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:41.980518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.980479 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:41.980518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.980507 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:41.998196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:41.998175 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm"] Oct 30 19:28:42.003783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.003763 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.013219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.013198 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ce69f3-openshift-default-dockercfg-ktjvt\"" Oct 30 19:28:42.016337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.016313 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm"] Oct 30 19:28:42.019242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.019219 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp"] Oct 30 19:28:42.024588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.024569 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.042988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.042958 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp"] Oct 30 19:28:42.065536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065514 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.065638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065546 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.065638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065576 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.065744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065722 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.065798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065778 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.065922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065903 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.065984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.065950 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgc8l\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.066036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066003 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xdk\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.066075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066065 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.066108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066086 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.066145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066106 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.066145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066126 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.066210 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066177 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.066210 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066192 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.066286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066212 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.066286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066253 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.066386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066366 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.066451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.066432 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.167425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167394 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167437 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgc8l\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.167598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167460 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xdk\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167526 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167551 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167581 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167613 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167641 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167668 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167714 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167737 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167799 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167823 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.167918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.167913 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168011 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168018 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168070 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168111 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168142 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168177 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.168323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168238 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.168659 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168419 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.168659 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.168598 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.169271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.169242 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.169400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.169291 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.169587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.169566 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.169734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.169620 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.170119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.170097 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.170472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.170444 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.171217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.171184 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.172183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.172157 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.172285 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.172263 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.177380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.177356 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xdk\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.177738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.177712 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgc8l\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.179412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.179389 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.180518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.180496 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.190941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.190920 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:42.330783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.330693 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:42.356432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.356405 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:42.501465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.501422 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" event={"ID":"45155a92-257c-47b0-a18e-38bff73f2120","Type":"ContainerStarted","Data":"b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c"} Oct 30 19:28:42.501647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.501479 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" event={"ID":"45155a92-257c-47b0-a18e-38bff73f2120","Type":"ContainerStarted","Data":"90508dc5a740d4d02a8beeddf0242bd337e0adfc2f10295ab4367a58509e0a4c"} Oct 30 19:28:42.630258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.630206 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podStartSLOduration=1.630186289 podStartE2EDuration="1.630186289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:42.536672363 +0000 UTC m=+692.494491627" watchObservedRunningTime="2026-10-30 19:28:42.630186289 +0000 UTC m=+692.588005477" Oct 30 19:28:42.632492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.632463 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm"] Oct 30 19:28:42.632611 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:42.632502 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc182ca0d_7923_47fd_9a77_1d468c8ece56.slice/crio-8f4019f4d8123632ddf2e2cf61cfe13110010d4d2151718c6d45a8eedcc47044 WatchSource:0}: Error finding container 8f4019f4d8123632ddf2e2cf61cfe13110010d4d2151718c6d45a8eedcc47044: Status 404 returned error can't find the container with id 8f4019f4d8123632ddf2e2cf61cfe13110010d4d2151718c6d45a8eedcc47044 Oct 30 19:28:42.635159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.635129 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:42.635258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.635188 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:42.635258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.635218 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:42.655611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.655589 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp"] Oct 30 19:28:42.655683 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:42.655590 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f78ad1_02ff_4db7_90bf_46cf04ea1bd6.slice/crio-42a50f666de15cb423a5ba76914d1eb7899db6c76860cf5ec9db93dc145e6ca9 WatchSource:0}: Error finding container 42a50f666de15cb423a5ba76914d1eb7899db6c76860cf5ec9db93dc145e6ca9: Status 404 returned error can't find the container with id 42a50f666de15cb423a5ba76914d1eb7899db6c76860cf5ec9db93dc145e6ca9 Oct 30 19:28:42.658184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.658156 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:42.658264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.658231 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:42.658350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.658275 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:42.696007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:42.695941 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:43.039929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.039890 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:43.328971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.328273 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:43.531887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.531526 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" event={"ID":"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6","Type":"ContainerStarted","Data":"981b3d088539f6d42a39a67bdc05d9a09f1dbc8b37deff5b4004eba0f71e094d"} Oct 30 19:28:43.531887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.531666 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" event={"ID":"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6","Type":"ContainerStarted","Data":"42a50f666de15cb423a5ba76914d1eb7899db6c76860cf5ec9db93dc145e6ca9"} Oct 30 19:28:43.566886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.554746 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" event={"ID":"c182ca0d-7923-47fd-9a77-1d468c8ece56","Type":"ContainerStarted","Data":"d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57"} Oct 30 19:28:43.566886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.554800 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" event={"ID":"c182ca0d-7923-47fd-9a77-1d468c8ece56","Type":"ContainerStarted","Data":"8f4019f4d8123632ddf2e2cf61cfe13110010d4d2151718c6d45a8eedcc47044"} Oct 30 19:28:43.566886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.560776 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podStartSLOduration=2.560760126 podStartE2EDuration="2.560760126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:43.558152271 +0000 UTC m=+693.515971459" watchObservedRunningTime="2026-10-30 19:28:43.560760126 +0000 UTC m=+693.518579318" Oct 30 19:28:43.587015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.586891 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podStartSLOduration=2.586847541 podStartE2EDuration="2.586847541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:43.585830601 +0000 UTC m=+693.543649790" watchObservedRunningTime="2026-10-30 19:28:43.586847541 +0000 UTC m=+693.544666729" Oct 30 19:28:43.711975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.707977 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:28:43.711975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:43.708045 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:28:44.173127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.173023 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:44.345891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.334150 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:44.345891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.341982 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:28:44.345891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.342044 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:28:44.371894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.365144 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:44.371894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.365293 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:28:44.371894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.365338 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:28:44.694898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.694217 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:28:44.694898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:44.694273 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:28:45.356888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.356169 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:28:45.356888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.356240 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:28:45.358795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.358705 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:28:45.358795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.358748 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:28:45.422889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.419752 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:45.713889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.699278 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:28:45.713889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:45.699334 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:28:46.334887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.331383 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:28:46.334887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.331442 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:28:46.371304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.361250 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:28:46.371304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.361302 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:28:46.420714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.420538 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:46.730889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.710257 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:28:46.730889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:46.710318 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:28:47.341007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.334283 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:28:47.341007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.334345 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:28:47.358292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.358259 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:28:47.358525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.358498 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:28:47.710887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.705268 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:28:47.710887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.705330 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:28:47.733399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:47.733330 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:48.340886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.340247 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:28:48.340886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.340304 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:28:48.357027 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.356994 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:28:48.357170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.357051 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:28:48.584460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.584393 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:48.700075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.693370 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:28:48.700075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:48.693463 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 19:28:49.331488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.331440 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:28:49.332006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.331510 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:28:49.356900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.356826 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:28:49.357086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.356898 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:28:49.433625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.433587 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:49.700010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.699978 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:49.700333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.700297 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:49.701395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:49.701374 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:28:50.145471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.145395 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:50.335806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.335778 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:50.336313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.336174 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:50.336951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.336932 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:28:50.361265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.361244 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:50.361513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.361493 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:50.362055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:50.362039 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:28:50.883288 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:28:50.883252 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:28:51.030370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:51.030338 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:51.731122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:51.731090 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:51.845033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:51.845003 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:52.596888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:52.588295 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.436529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:53.436498 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.768256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:53.768170 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:28:54.274222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:54.274186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:54.779906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:54.779832 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:55.849928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:55.845095 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:56.382916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.382592 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:56.507889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.506988 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:56.607981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.607945 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5"] Oct 30 19:28:56.722943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.721929 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:56.821922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.809218 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:56.843887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.811140 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:56.920186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:56.920131 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.064384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.064269 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.229889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.219208 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5"] Oct 30 19:28:57.241033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.240995 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.241261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.241239 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.265048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.265021 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.274915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.270458 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.275320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.275293 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.275537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.275519 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.275730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.275711 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.275912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.275894 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.276313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.276294 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.280175 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.245939 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf"] Oct 30 19:28:57.326047 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.325983 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf"] Oct 30 19:28:57.326312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.326297 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.328716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.328695 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn"] Oct 30 19:28:57.366192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.366170 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.380955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.380924 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.381480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.381459 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.381683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.381663 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.382556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.382536 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.394138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394116 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.394341 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394323 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.394491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394471 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.394603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394587 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.394719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394704 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.394832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394818 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcp4\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.395005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394977 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.395173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395157 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.395292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395276 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.395407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395393 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.395564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395544 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.395714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395698 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.395836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395819 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.395993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.395970 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.396197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.382440 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.396343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.382936 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:28:57.396720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.394036 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.417885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.397259 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.420949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.398659 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.421092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.399349 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45"] Oct 30 19:28:57.421997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.421977 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:57.422155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.422132 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.423187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.416171 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.433530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.433500 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.446279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.446251 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.449954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.448126 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:28:57.450104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.450087 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn"] Oct 30 19:28:57.454693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.454674 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.457139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.457081 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.461975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.457768 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.497274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.497241 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.497529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.497507 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75g6\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.497679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.497651 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.497850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.497831 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.498017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.497999 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.498144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498127 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.498249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498234 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.498352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498338 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.498472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498458 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.498572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498559 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.498674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498661 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.498776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.498762 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.660187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.524731 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45"] Oct 30 19:28:57.660187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.590970 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.660187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.591564 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:28:57.660755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.660723 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.660979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.660959 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcp4\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.665952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.665925 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.666152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.666133 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.670148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.666278 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.670311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.670288 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.670493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.670475 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.684184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684159 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.684398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684380 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.684512 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684498 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.684637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684621 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.684750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684737 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.684853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684838 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9hz\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.685003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.684986 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.685114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.685097 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.708974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.708944 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.709470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.709446 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.710347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.710323 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.731391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.710767 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.754889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.738579 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:57.754889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.741344 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.789583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.779876 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:57.789583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.786940 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:28:57.835050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.834955 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.934708 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.934835 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75g6\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.934899 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.934963 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.934999 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935046 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935105 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935129 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935158 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935183 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935230 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935330 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935366 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935423 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935497 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935528 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9hz\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935563 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.935589 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.936081 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.963937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.937083 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:57.987885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:57.977764 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:58.004789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.004748 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:58.005499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.005383 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:28:58.033081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.028236 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.033081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.032727 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:58.051946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.047218 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:28:58.134957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.071759 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.134957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.072207 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.145841 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.146572 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.158886 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.184098 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.185705 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.220342 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.323004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.314797 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.341141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.337574 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75g6\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.396369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.394718 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.397558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.397533 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.440355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.440298 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:28:58.440355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.440356 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:28:58.484219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.483113 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.484219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.483719 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.484219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.483982 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.484219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.484180 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.541440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.541389 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.543310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.543177 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:58.544141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.544114 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcp4\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:58.545069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.544547 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9hz\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.545651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.545609 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:58.553149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.553122 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:28:58.561703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.556157 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:28:58.580793 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.580717 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:28:58.649102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.649001 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:58.657328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.657230 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:58.874537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.874501 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5"] Oct 30 19:28:58.877016 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:58.876993 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f2142_4e28_4b9d_bcd0_03799519cbe7.slice/crio-1fd5b1b5460bbed06f701a3487bfb6c1cf707841f570c7feab74b7a3dbf8ee1a WatchSource:0}: Error finding container 1fd5b1b5460bbed06f701a3487bfb6c1cf707841f570c7feab74b7a3dbf8ee1a: Status 404 returned error can't find the container with id 1fd5b1b5460bbed06f701a3487bfb6c1cf707841f570c7feab74b7a3dbf8ee1a Oct 30 19:28:58.882627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.882583 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.882740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.882693 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.882803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.882760 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.920311 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:28:58.920283 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b47461_0713_4d2d_ac1a_4713d1bcfe76.slice/crio-41e88f52cc6f8499406d4d06e9c06311c15586f449063be2bd78d1dbef4b761f WatchSource:0}: Error finding container 41e88f52cc6f8499406d4d06e9c06311c15586f449063be2bd78d1dbef4b761f: Status 404 returned error can't find the container with id 41e88f52cc6f8499406d4d06e9c06311c15586f449063be2bd78d1dbef4b761f Oct 30 19:28:58.921783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.921759 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf"] Oct 30 19:28:58.923234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.923201 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.923413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.923383 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.923487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.923431 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.961036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:58.961006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:59.035265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.035171 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45"] Oct 30 19:28:59.041656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.041573 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn"] Oct 30 19:28:59.055912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.050025 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.055912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.050108 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.055912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.050154 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.118906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.112464 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.118906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.112566 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.118906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.112606 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.589131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.589092 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" event={"ID":"db9eeec3-9f96-483e-ae46-54ff7776dd06","Type":"ContainerStarted","Data":"87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f"} Oct 30 19:28:59.589323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.589156 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" event={"ID":"db9eeec3-9f96-483e-ae46-54ff7776dd06","Type":"ContainerStarted","Data":"01f4966ea7cb88364538086cb375a27df4fcb5e3c925f446dfda2075d3c58dc4"} Oct 30 19:28:59.594448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.594417 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" event={"ID":"40b47461-0713-4d2d-ac1a-4713d1bcfe76","Type":"ContainerStarted","Data":"181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432"} Oct 30 19:28:59.594570 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.594473 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" event={"ID":"40b47461-0713-4d2d-ac1a-4713d1bcfe76","Type":"ContainerStarted","Data":"41e88f52cc6f8499406d4d06e9c06311c15586f449063be2bd78d1dbef4b761f"} Oct 30 19:28:59.596944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.596920 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" event={"ID":"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab","Type":"ContainerStarted","Data":"e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2"} Oct 30 19:28:59.597049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.596964 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" event={"ID":"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab","Type":"ContainerStarted","Data":"3c17f3847de2578fd1f8af16e6167ca1d52f2387f80e629374f6f5eb524ed493"} Oct 30 19:28:59.600163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.600140 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" event={"ID":"342f2142-4e28-4b9d-bcd0-03799519cbe7","Type":"ContainerStarted","Data":"dbefef822062e45f1eb7dfebefe50fd288cea05492990d7ef63556fc0bcb7756"} Oct 30 19:28:59.600290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.600178 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" event={"ID":"342f2142-4e28-4b9d-bcd0-03799519cbe7","Type":"ContainerStarted","Data":"1fd5b1b5460bbed06f701a3487bfb6c1cf707841f570c7feab74b7a3dbf8ee1a"} Oct 30 19:28:59.627608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.627551 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podStartSLOduration=3.627533212 podStartE2EDuration="3.627533212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:59.623722986 +0000 UTC m=+709.581542176" watchObservedRunningTime="2026-10-30 19:28:59.627533212 +0000 UTC m=+709.585352403" Oct 30 19:28:59.658247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.658217 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:28:59.658376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.658264 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:28:59.661664 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.661621 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podStartSLOduration=3.661608949 podStartE2EDuration="3.661608949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:59.660505652 +0000 UTC m=+709.618324855" watchObservedRunningTime="2026-10-30 19:28:59.661608949 +0000 UTC m=+709.619428136" Oct 30 19:28:59.702884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.702792 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podStartSLOduration=3.702768197 podStartE2EDuration="3.702768197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:59.694360708 +0000 UTC m=+709.652179897" watchObservedRunningTime="2026-10-30 19:28:59.702768197 +0000 UTC m=+709.660587387" Oct 30 19:28:59.722853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.722814 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podStartSLOduration=3.722804504 podStartE2EDuration="3.722804504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:59.721344625 +0000 UTC m=+709.679163814" watchObservedRunningTime="2026-10-30 19:28:59.722804504 +0000 UTC m=+709.680623691" Oct 30 19:28:59.751551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.751526 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:28:59.752728 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.752705 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:28:59.752813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:28:59.752751 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:00.068888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.068490 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:00.503984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.494177 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:00.641884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.635076 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:29:00.774966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.738502 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.859524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.859487 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:00.859990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.739146 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.860183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.860159 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:00.860315 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.739187 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.860411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.860395 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:00.860501 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.751509 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.860593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:00.860579 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:01.205889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.203980 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.337886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.336975 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.575941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.575007 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.599887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.583186 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.599887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.583234 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:01.599887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.589467 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:01.665965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.665928 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.666257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.666221 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:01.677442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.666842 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.677702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.677667 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:01.767933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.766469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:01.767933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.767682 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.767933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.767731 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:01.806925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.803149 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.806925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:01.803509 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:29:02.213007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.212656 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.226893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.222647 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:29:02.359608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.359563 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.598190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.598039 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.602901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.600958 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:29:02.633962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.616733 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.633962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.616785 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:02.658111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.658080 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.658459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.658432 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:02.658933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.658353 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.659107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.659086 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:02.761875 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.761824 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.801908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.795464 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:02.828878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.828828 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.829519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.829500 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:29:02.853384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.828780 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.855036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:02.853909 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:29:03.231887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.231341 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:29:03.354011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.353963 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.403889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.374006 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.403889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.374597 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:29:03.537961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.533401 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:03.617920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.609008 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:03.617920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.609766 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.617920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.611250 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.617920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.611285 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:03.675044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.674988 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.675307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.675279 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:03.679962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.676978 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.687885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.680136 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:03.779918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.777035 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.779918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.777093 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:03.856941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.832954 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.857404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:03.857365 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.270006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.233952 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.362885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.356951 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 19:29:04.363207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.363177 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:29:04.368286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.368262 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:29:04.454004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.453964 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.613081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.612978 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.613968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.613942 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:29:04.670893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.628511 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:04.681976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.646415 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.681976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.675283 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:04.681976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.664793 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.681976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.675342 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:04.681976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.665986 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.681976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.675384 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:04.781890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.771623 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.781890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.771682 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:04.833298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.833256 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.865755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:04.865657 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.299888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.288425 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.335697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.335650 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.369960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.369919 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:29:05.500913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.478004 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.597931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.597550 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:05.597931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.597624 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:05.643883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.625939 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:05.669439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.669402 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:05.706907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.674934 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:05.706907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.675036 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.706907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.669704 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:05.706907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.675318 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:05.819049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.818992 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:05.917719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.884463 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.917719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.915055 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.942291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.942168 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:05.975887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:05.972916 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:06.007375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.007328 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:29:06.305821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.305775 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.336607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.336556 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.385888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.370966 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.527884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.527334 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.598379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.598274 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:06.654899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.651235 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:06.654899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.636142 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.661994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.661963 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:06.662814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.662783 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:06.663073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.662209 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:06.663196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.663177 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:06.677885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.676769 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.758886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.753192 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.758886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.753534 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 19:29:06.758886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.753629 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:06.758886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.753657 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:06.786304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.786273 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:29:06.890062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.889958 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.977898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:06.947462 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:29:07.110915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.107242 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:29:07.120884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.113154 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:29:07.120884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.114178 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:29:07.264571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.264533 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:29:07.289984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.289952 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:29:07.360529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.360491 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:07.582461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.582377 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:29:07.582461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.582429 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 19:29:07.660240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.660193 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:29:07.660433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.660263 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 19:29:07.660684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.660636 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:29:07.660928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.660898 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:29:07.751314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.751280 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:29:07.751498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:07.751343 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 19:29:08.362618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.362547 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:08.641165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.641083 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:29:08.646141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.643575 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:29:08.648359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.648339 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:29:08.721944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.721912 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:29:08.722623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.722594 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:29:08.723215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.723180 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:29:08.755836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.755806 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:29:08.756125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.756106 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:29:08.756745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.756726 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:29:08.988239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.988194 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:29:08.989306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.989282 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:29:08.997857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:08.990189 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:29:09.648068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:09.648042 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:10.225902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:10.225851 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:10.766872 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:10.766841 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:11.556513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.556454 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:11.751646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.751612 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp"] Oct 30 19:29:11.757385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.757361 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.774745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.774720 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp"] Oct 30 19:29:11.784358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784334 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.784467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784381 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.784513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784479 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.784683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784655 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.784783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784766 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.784821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784807 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.784883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.784851 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.785023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.785005 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfbmm\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.785092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.785059 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886363 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfbmm\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886405 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886451 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886476 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886749 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886891 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.886992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886918 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.887069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.886998 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.887069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.887053 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.887174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.887104 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.887174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.887126 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.887275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.887261 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.887514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.887489 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.888195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.888137 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.889845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.889822 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.891332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.891204 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.896339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.896314 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfbmm\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:11.897072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:11.897051 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:12.081334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.081306 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:12.274187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.274155 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm"] Oct 30 19:29:12.279391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.279372 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.291108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.291082 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm"] Oct 30 19:29:12.293243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293218 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293259 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293354 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293490 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293473 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293559 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293594 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293682 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293637 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlntw\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293736 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.293822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.293777 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.385272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.385250 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp"] Oct 30 19:29:12.385371 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:29:12.385354 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaafa3a2_1a01_4a5f_ba25_e1e48a5e4dbe.slice/crio-4a1d669b529d3658430eea72d3aefcc3f122c1806819e6438ab17bf5f48d0449 WatchSource:0}: Error finding container 4a1d669b529d3658430eea72d3aefcc3f122c1806819e6438ab17bf5f48d0449: Status 404 returned error can't find the container with id 4a1d669b529d3658430eea72d3aefcc3f122c1806819e6438ab17bf5f48d0449 Oct 30 19:29:12.388382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.388348 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:12.388488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.388425 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:12.388488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.388471 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:12.395112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395056 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395109 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395133 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlntw\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395180 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395204 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395653 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395699 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395750 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395881 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.395978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.395888 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.396274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.396125 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.396274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.396257 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.396432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.396414 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.396791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.396770 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.398077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.398060 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.398505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.398481 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.408239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.408200 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.409855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.409833 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlntw\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.447332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.447305 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" event={"ID":"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe","Type":"ContainerStarted","Data":"4a1d669b529d3658430eea72d3aefcc3f122c1806819e6438ab17bf5f48d0449"} Oct 30 19:29:12.598072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.598001 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:12.604008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.602749 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:12.900982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.900949 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm"] Oct 30 19:29:12.905277 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:29:12.905251 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34097a6a_eb26_4df5_aaec_3ad0126a0f40.slice/crio-bbbae329d7999be4c217e59a84b31d3586f23b2a6f03ec312670c1837a1c8194 WatchSource:0}: Error finding container bbbae329d7999be4c217e59a84b31d3586f23b2a6f03ec312670c1837a1c8194: Status 404 returned error can't find the container with id bbbae329d7999be4c217e59a84b31d3586f23b2a6f03ec312670c1837a1c8194 Oct 30 19:29:12.908450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.907921 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:12.908450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.908002 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:12.908450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:12.908072 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:13.456697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.456660 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" event={"ID":"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe","Type":"ContainerStarted","Data":"b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a"} Oct 30 19:29:13.458505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.458474 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" event={"ID":"34097a6a-eb26-4df5-aaec-3ad0126a0f40","Type":"ContainerStarted","Data":"9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0"} Oct 30 19:29:13.458666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.458515 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" event={"ID":"34097a6a-eb26-4df5-aaec-3ad0126a0f40","Type":"ContainerStarted","Data":"bbbae329d7999be4c217e59a84b31d3586f23b2a6f03ec312670c1837a1c8194"} Oct 30 19:29:13.481007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.480964 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podStartSLOduration=2.480950849 podStartE2EDuration="2.480950849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:13.48024316 +0000 UTC m=+723.438062349" watchObservedRunningTime="2026-10-30 19:29:13.480950849 +0000 UTC m=+723.438770036" Oct 30 19:29:13.484211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.484186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:13.544907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.535622 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podStartSLOduration=1.535598939 podStartE2EDuration="1.535598939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:13.516416913 +0000 UTC m=+723.474236099" watchObservedRunningTime="2026-10-30 19:29:13.535598939 +0000 UTC m=+723.493418129" Oct 30 19:29:13.605887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:13.605182 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:14.081437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:14.081399 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:14.082916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:14.082851 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:29:14.083088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:14.082923 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:29:14.523842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:14.523805 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:14.618890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:14.616082 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:29:14.618890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:14.616182 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:29:15.081752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:15.081714 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:29:15.082159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:15.081780 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:29:15.418952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:15.413334 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:15.612332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:15.612300 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:29:15.612606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:15.612580 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:29:16.081746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:16.081713 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:29:16.081953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:16.081786 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:29:16.379103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:16.379010 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:16.616886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:16.613967 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:29:16.616886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:16.614026 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:29:17.081998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:17.081967 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:29:17.082359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:17.082014 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:29:17.262122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:17.262093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:17.615889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:17.605551 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:29:17.615889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:17.605621 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:29:17.678228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:17.678190 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:18.082117 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:18.082089 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:29:18.082575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:18.082549 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:29:18.629477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:18.629437 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:29:18.630504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:18.630468 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:29:18.833147 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:18.833113 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:19.082378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:19.082342 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:29:19.082919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:19.082411 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:29:19.553550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:19.553511 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:19.616304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:19.608658 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:29:19.616304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:19.608715 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:29:20.086305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.086261 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:20.086814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.086717 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:20.087819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.087797 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:29:20.293920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.293889 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:20.614753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.614723 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:20.615209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.615189 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:20.615980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:20.615961 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:29:20.890164 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:29:20.890070 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:29:21.140158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:21.140123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:21.956051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:21.956018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:22.828511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:22.828472 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:23.657141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:23.657113 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:24.440592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:24.440560 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:25.157849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:25.157813 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:26.741887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:26.739566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:27.279887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.279826 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b"] Oct 30 19:29:27.298982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.298931 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:27.307443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.307418 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.318905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.311002 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b"] Oct 30 19:29:27.383108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.381542 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:27.417886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.414738 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.430887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.422020 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.438273 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.438245 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.438670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.438645 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8msn\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.465615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.465589 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.465893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.465852 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.466149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.466130 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.466356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.466339 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.466528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.466511 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.572788 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.572847 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.572911 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573014 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8msn\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573050 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573079 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573116 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573181 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573227 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.573748 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.574330 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.579950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.574570 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.638703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.638657 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:27.640808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.640780 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.641624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.641602 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.820905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.801976 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8msn\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.854277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.837217 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.854481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.842507 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:27.855304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.854149 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.880525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.880490 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:27.881666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.881633 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:27.976886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:27.957533 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:28.028013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.002522 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:28.120931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.117690 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:28.328895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.314949 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:28.328895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.316669 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:29:28.392615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.390370 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:28.392615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.391819 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:29:28.710885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.707495 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:28.710885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.710560 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:29:28.958940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.881085 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:28.958940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.881725 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:29:29.014346 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:28.968446 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:29:29.014940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:29.014902 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:29.195279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:29.189253 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:29:29.432937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:29.432638 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:29:29.657375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:29.638340 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:30.029490 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:29:30.029450 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1f2674_c231_4bf7_8d3c_d11051d0b9bf.slice/crio-43b08dfdd1a484715547876cdf82b1ba917d2ec97dd0b59c641a22b20b871907 WatchSource:0}: Error finding container 43b08dfdd1a484715547876cdf82b1ba917d2ec97dd0b59c641a22b20b871907: Status 404 returned error can't find the container with id 43b08dfdd1a484715547876cdf82b1ba917d2ec97dd0b59c641a22b20b871907 Oct 30 19:29:30.040753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.040670 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:30.048948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.045891 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:29:30.048948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.046303 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7" podUID="5d55749f-62cd-4021-b995-5e9245cdbaad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:30.048948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.047519 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v5vhh" podUID="04daab4c-1b99-4724-8649-630cdb2b4bc4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.37:50052\" within 1s: context deadline exceeded" Oct 30 19:29:30.048948 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:29:30.047721 6514 logging.go:55] [core] [Channel #866 SubChannel #867]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.37:50052", ServerName: "10.134.0.37:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.37:50052: operation was canceled" Oct 30 19:29:30.077858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.077150 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:29:30.080055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.080026 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:30.103300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.103262 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.113038 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.148965 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.148998 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b"] Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.149076 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.120818 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg" podUID="6dda9c25-033d-4e68-bcf8-ba726c31024e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.125414 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.108709 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.149514 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:30.149819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.149562 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:30.179763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.179737 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:29:30.479394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:30.479356 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:31.139508 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:31.139466 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" event={"ID":"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf","Type":"ContainerStarted","Data":"9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d"} Oct 30 19:29:31.140041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:31.139536 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" event={"ID":"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf","Type":"ContainerStarted","Data":"43b08dfdd1a484715547876cdf82b1ba917d2ec97dd0b59c641a22b20b871907"} Oct 30 19:29:31.166565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:31.166520 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podStartSLOduration=4.166507804 podStartE2EDuration="4.166507804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:31.164594756 +0000 UTC m=+741.122413944" watchObservedRunningTime="2026-10-30 19:29:31.166507804 +0000 UTC m=+741.124326990" Oct 30 19:29:31.580814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:31.580776 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:32.015889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:32.010126 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:32.017207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:32.017183 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:29:32.017318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:32.017227 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:29:32.467520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:32.467467 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:33.014717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:33.010176 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:29:33.014717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:33.010235 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:29:33.672818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:33.672786 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:34.016347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:34.016313 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:29:34.016552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:34.016374 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:29:34.555248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:34.555220 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:35.016561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:35.016524 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:29:35.017208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:35.017168 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:29:35.460334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:35.460304 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:36.016004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:36.015967 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:29:36.016198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:36.016030 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:29:36.424672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:36.424573 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:37.007149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:37.007115 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:29:37.007327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:37.007168 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:29:37.384286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:37.384192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:38.011416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:38.011383 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:38.011837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:38.011819 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:38.012658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:38.012639 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:29:38.071980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:38.071939 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:38.745001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:38.744970 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:39.535742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:39.535714 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:40.253252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:40.253222 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:40.961167 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:40.961136 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:41.554581 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.554548 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs"] Oct 30 19:29:41.560530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.560507 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.573638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.573613 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs"] Oct 30 19:29:41.646536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646501 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646555 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646584 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646607 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646717 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lhc\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646764 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646781 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646804 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.646893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.646829 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.665355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.665326 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj"] Oct 30 19:29:41.670543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.670522 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.693635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.693611 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj"] Oct 30 19:29:41.748201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748177 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2nk\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.748348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748213 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.748348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748252 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.748438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748383 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.748484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748434 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.748484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748478 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748498 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748516 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748530 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748632 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.748738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748664 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lhc\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748703 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748752 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748787 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748820 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748838 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.748930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748893 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.749206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748959 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.749206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.748977 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.749206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.749101 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.749206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.749140 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.749401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.749337 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.749441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.749408 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.751267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.751243 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.751668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.751648 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.766031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.766010 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.772880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.772846 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lhc\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.850703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.850647 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.850789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.850706 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.850789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.850752 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.850882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.850831 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2nk\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.850996 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851047 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851100 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851161 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851180 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851232 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851293 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851383 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851443 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.851837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.851812 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.852842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.852817 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.880107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.873365 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.895906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.882135 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:41.895906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.890728 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.928881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.920844 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2nk\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:41.940164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.940083 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:41.998974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:41.998940 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:42.265076 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:29:42.265046 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984bd045_8081_46a7_a3a2_897b4210b861.slice/crio-dd9a2dbe3914e29facf7bb96987267924f453311b2fa3c0d06a5fdf0dfbd45a9 WatchSource:0}: Error finding container dd9a2dbe3914e29facf7bb96987267924f453311b2fa3c0d06a5fdf0dfbd45a9: Status 404 returned error can't find the container with id dd9a2dbe3914e29facf7bb96987267924f453311b2fa3c0d06a5fdf0dfbd45a9 Oct 30 19:29:42.266391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.266360 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs"] Oct 30 19:29:42.267382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.267346 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:42.267465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.267417 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:42.267465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.267456 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:42.323926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.321584 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj"] Oct 30 19:29:42.323926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.323855 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:42.324152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.323994 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:42.324152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.324077 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:42.738224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:42.738147 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:43.250369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.250334 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" event={"ID":"80f28040-03ae-4491-b1db-2f289961c491","Type":"ContainerStarted","Data":"6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa"} Oct 30 19:29:43.250748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.250387 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" event={"ID":"80f28040-03ae-4491-b1db-2f289961c491","Type":"ContainerStarted","Data":"ae4350a03971f2b7c4d7bc4af5698f8cd72f5fd28af1546303e4ac22ccc693be"} Oct 30 19:29:43.252818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.252790 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" event={"ID":"984bd045-8081-46a7-a3a2-897b4210b861","Type":"ContainerStarted","Data":"24315fe16050401907c852ca39f94802f4a7b2fd18b9872cf3297f81171fe6fe"} Oct 30 19:29:43.252968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.252835 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" event={"ID":"984bd045-8081-46a7-a3a2-897b4210b861","Type":"ContainerStarted","Data":"dd9a2dbe3914e29facf7bb96987267924f453311b2fa3c0d06a5fdf0dfbd45a9"} Oct 30 19:29:43.277039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.276994 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podStartSLOduration=2.276981485 podStartE2EDuration="2.276981485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:43.274347778 +0000 UTC m=+753.232166989" watchObservedRunningTime="2026-10-30 19:29:43.276981485 +0000 UTC m=+753.234800673" Oct 30 19:29:43.314888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.314713 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podStartSLOduration=2.314692815 podStartE2EDuration="2.314692815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:43.313167376 +0000 UTC m=+753.270986564" watchObservedRunningTime="2026-10-30 19:29:43.314692815 +0000 UTC m=+753.272512003" Oct 30 19:29:43.774934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.767776 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:43.901888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.897323 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:43.904893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.903966 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:29:43.904893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:43.904024 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:29:44.014888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:44.014341 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:44.024887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:44.019962 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:29:44.024887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:44.020014 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:29:44.744886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:44.744644 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:44.895374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:44.894925 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:29:44.895374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:44.894979 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:29:45.025949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:45.021230 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:29:45.025949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:45.021289 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:29:45.636515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:45.636482 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:45.908938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:45.900141 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:29:45.908938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:45.900198 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:29:46.004238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:46.004182 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:29:46.004404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:46.004257 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:29:46.618896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:46.618293 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:46.895327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:46.895216 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:29:46.895584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:46.895531 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:29:47.003279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:47.003243 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:29:47.003838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:47.003807 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:29:47.657322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:47.657287 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:47.911950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:47.908909 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:29:47.911950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:47.908966 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:29:48.000213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:48.000178 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.000481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:48.000442 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:29:48.723179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:48.723146 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:48.892098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:48.892061 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.892312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:48.892118 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:29:49.000467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:49.000368 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:29:49.000467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:49.000444 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:29:49.749816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:49.749753 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:49.961308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:49.961280 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:49.962447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:49.962425 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:49.963122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:49.963102 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:29:50.004452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:50.004392 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:50.004677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:50.004659 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:50.005283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:50.005265 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:29:50.657118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:50.657080 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:50.888104 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:29:50.888067 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:29:51.661560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:51.661530 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.451545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:52.451504 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:53.371456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:53.371415 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:54.084948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:54.084907 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:54.750227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:54.750197 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.434738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:55.434700 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.788854 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:55.788768 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:29:56.351423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:56.351386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.860526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:57.860482 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:57.914717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:57.897938 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.101392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.081184 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.197886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.196041 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:29:58.220887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.200772 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.307888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.280235 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.658237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.658136 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.891886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.888411 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.891886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.888893 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:29:58.908938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.899098 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:58.908938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:58.899658 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:29:59.103890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.089021 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.103890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.089758 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:29:59.235888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.232036 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.261174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.234842 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:29:59.397888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.297075 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:29:59.397888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.297381 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:29:59.455623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.408024 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:59.455623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.408425 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:29:59.687948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.666821 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.714017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.713981 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:29:59.944738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.893941 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:29:59.945522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:29:59.911260 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:00.095040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.094961 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:00.402944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.383314 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:00.430318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.409173 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.446005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.429954 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:00.730100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.694141 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.992939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.946758 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.992939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.967961 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 19:30:00.992939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:00.968444 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:01.144944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.025552 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:01.144944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.117553 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:01.465893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.454653 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.465893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.464502 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:01.484898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.473221 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:01.741079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.731092 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.961888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.953991 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.265917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:01.979183 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.265917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.128139 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.479889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.478846 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.487888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.486109 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:02.500985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.499439 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:02.768008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.767954 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.950889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.809921 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6"] Oct 30 19:30:02.999442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.999407 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7"] Oct 30 19:30:02.999898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:02.999851 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.058360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.057059 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6"] Oct 30 19:30:03.058360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.057113 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7"] Oct 30 19:30:03.058360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.057213 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.063529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.063502 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:30:03.089266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.089121 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.089406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.089290 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.089459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.089423 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64tf\" (UniqueName: \"kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.092144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.089590 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrxd\" (UniqueName: \"kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd\") pod \"maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.201796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.190732 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.201796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.190859 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64tf\" (UniqueName: \"kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.201796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.191018 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrxd\" (UniqueName: \"kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd\") pod \"maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.201796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.191128 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.201796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.191998 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.201796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.192619 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.246454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.246422 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64tf\" (UniqueName: \"kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.247477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.247451 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrxd\" (UniqueName: \"kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd\") pod \"maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.256267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.256219 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.267223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.267191 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.267985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.267958 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.281313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.281288 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:30:03.447497 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.446014 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:03.472612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.472568 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:03.488109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.482487 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.493045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.493011 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.516729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.509011 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.792943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:03.792253 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:04.288495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.285801 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:04.288495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.286120 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:04.495054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.494994 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:04.560696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.560600 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:04.588833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.556719 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:04.657174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.657126 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:30:04.657543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.657465 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:30:04.676897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.676051 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:30:04.682337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.682311 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:30:04.683892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.683123 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:04.690959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.688998 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:30:04.773908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.760177 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7"] Oct 30 19:30:04.892999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:04.892966 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6"] Oct 30 19:30:04.895359 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:04.895319 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fc7cd4_60ad_40d6_982c_7f714d897e45.slice/crio-ff6cb3fa66f6e9be9971227f3a953c04b032fa987da5db7df0161578739786d7 WatchSource:0}: Error finding container ff6cb3fa66f6e9be9971227f3a953c04b032fa987da5db7df0161578739786d7: Status 404 returned error can't find the container with id ff6cb3fa66f6e9be9971227f3a953c04b032fa987da5db7df0161578739786d7 Oct 30 19:30:05.337198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.337154 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:05.674630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.674516 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerStarted","Data":"1df493fe622e9fc9718774dd8ecfbeb7e8ab78cbd22fade7b5b633a979903ac4"} Oct 30 19:30:05.674630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.674582 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerStarted","Data":"ff6cb3fa66f6e9be9971227f3a953c04b032fa987da5db7df0161578739786d7"} Oct 30 19:30:05.676475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.676442 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerStarted","Data":"fb3cf54c14dba6532506a1b2dadff9777234e023d9aad796c5da6d81892376cb"} Oct 30 19:30:05.676595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.676495 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerStarted","Data":"97350229ac43b3facb454f290b13b25e4df768715f0a4da44cb1101a95c3196a"} Oct 30 19:30:05.703724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.703683 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" podStartSLOduration=5.70366994 podStartE2EDuration="5.70366994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.702556804 +0000 UTC m=+775.660375992" watchObservedRunningTime="2026-10-30 19:30:05.70366994 +0000 UTC m=+775.661489129" Oct 30 19:30:05.729284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.729240 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" podStartSLOduration=5.729228212 podStartE2EDuration="5.729228212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.726654176 +0000 UTC m=+775.684473364" watchObservedRunningTime="2026-10-30 19:30:05.729228212 +0000 UTC m=+775.687047444" Oct 30 19:30:05.848393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.848362 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr"] Oct 30 19:30:05.852856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.852835 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:05.873199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.873174 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr"] Oct 30 19:30:05.920576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:05.920547 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:06.003171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.003142 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.003305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.003274 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.003470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.003451 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwl2\" (UniqueName: \"kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.104965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.104937 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.105110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.105041 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwl2\" (UniqueName: \"kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.105188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.105177 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.105835 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.105811 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.107265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.107241 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.113912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.113892 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwl2\" (UniqueName: \"kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2\") pod \"maas-api-revoke-sub-5043d15a-5hfbr\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.176853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.176827 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:06.247154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.247084 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.454257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.454228 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr"] Oct 30 19:30:06.455368 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:06.455340 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140956c2_0f99_4931_aec6_021b0b16f136.slice/crio-580b091c52f293ab30e24f69f3cdb061feae6e95022c1f276c0d80434bad317b WatchSource:0}: Error finding container 580b091c52f293ab30e24f69f3cdb061feae6e95022c1f276c0d80434bad317b: Status 404 returned error can't find the container with id 580b091c52f293ab30e24f69f3cdb061feae6e95022c1f276c0d80434bad317b Oct 30 19:30:06.684700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.684665 6514 generic.go:356] "Generic (PLEG): container finished" podID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerID="fb3cf54c14dba6532506a1b2dadff9777234e023d9aad796c5da6d81892376cb" exitCode=60 Oct 30 19:30:06.684847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.684754 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerDied","Data":"fb3cf54c14dba6532506a1b2dadff9777234e023d9aad796c5da6d81892376cb"} Oct 30 19:30:06.685063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.685043 6514 scope.go:122] "RemoveContainer" containerID="fb3cf54c14dba6532506a1b2dadff9777234e023d9aad796c5da6d81892376cb" Oct 30 19:30:06.686182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.686159 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" event={"ID":"140956c2-0f99-4931-aec6-021b0b16f136","Type":"ContainerStarted","Data":"580b091c52f293ab30e24f69f3cdb061feae6e95022c1f276c0d80434bad317b"} Oct 30 19:30:06.687989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.687963 6514 generic.go:356] "Generic (PLEG): container finished" podID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerID="1df493fe622e9fc9718774dd8ecfbeb7e8ab78cbd22fade7b5b633a979903ac4" exitCode=60 Oct 30 19:30:06.688106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.688022 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerDied","Data":"1df493fe622e9fc9718774dd8ecfbeb7e8ab78cbd22fade7b5b633a979903ac4"} Oct 30 19:30:06.688337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:06.688301 6514 scope.go:122] "RemoveContainer" containerID="1df493fe622e9fc9718774dd8ecfbeb7e8ab78cbd22fade7b5b633a979903ac4" Oct 30 19:30:07.121315 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.121292 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.588056 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.588027 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.697509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.697477 6514 generic.go:356] "Generic (PLEG): container finished" podID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerID="404349e16b9e5b6ab37911f2fda81e7e12b13ee37204dc5cd7f227f18bbfce75" exitCode=60 Oct 30 19:30:07.697661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.697557 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerDied","Data":"404349e16b9e5b6ab37911f2fda81e7e12b13ee37204dc5cd7f227f18bbfce75"} Oct 30 19:30:07.697661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.697623 6514 scope.go:122] "RemoveContainer" containerID="1df493fe622e9fc9718774dd8ecfbeb7e8ab78cbd22fade7b5b633a979903ac4" Oct 30 19:30:07.698005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.697978 6514 scope.go:122] "RemoveContainer" containerID="404349e16b9e5b6ab37911f2fda81e7e12b13ee37204dc5cd7f227f18bbfce75" Oct 30 19:30:07.698347 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:07.698324 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6_odh-ai-gateway-infra(62fc7cd4-60ad-40d6-982c-7f714d897e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" Oct 30 19:30:07.699851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.699829 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerStarted","Data":"28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744"} Oct 30 19:30:07.701542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.701522 6514 generic.go:356] "Generic (PLEG): container finished" podID="140956c2-0f99-4931-aec6-021b0b16f136" containerID="557783d7716400659ab0057bfe501c3b8f620a8aee4fd311928c78973e4164fd" exitCode=0 Oct 30 19:30:07.701628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.701562 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" event={"ID":"140956c2-0f99-4931-aec6-021b0b16f136","Type":"ContainerDied","Data":"557783d7716400659ab0057bfe501c3b8f620a8aee4fd311928c78973e4164fd"} Oct 30 19:30:07.851201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:07.851162 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.652471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.652439 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.752241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.752199 6514 scope.go:122] "RemoveContainer" containerID="404349e16b9e5b6ab37911f2fda81e7e12b13ee37204dc5cd7f227f18bbfce75" Oct 30 19:30:08.752651 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:08.752618 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6_odh-ai-gateway-infra(62fc7cd4-60ad-40d6-982c-7f714d897e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" Oct 30 19:30:08.757151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.757125 6514 generic.go:356] "Generic (PLEG): container finished" podID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerID="28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744" exitCode=60 Oct 30 19:30:08.758149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.757312 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerDied","Data":"28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744"} Oct 30 19:30:08.760214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.757744 6514 scope.go:122] "RemoveContainer" containerID="28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744" Oct 30 19:30:08.760911 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:08.760732 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7_odh-ai-gateway-infra(78df38a3-9842-40d2-80f3-4cbe84475c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" Oct 30 19:30:08.761538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.761163 6514 scope.go:122] "RemoveContainer" containerID="fb3cf54c14dba6532506a1b2dadff9777234e023d9aad796c5da6d81892376cb" Oct 30 19:30:08.937548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.937526 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:08.999479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.999451 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2\" (UniqueName: \"kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2\") pod \"140956c2-0f99-4931-aec6-021b0b16f136\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " Oct 30 19:30:08.999613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.999556 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca\") pod \"140956c2-0f99-4931-aec6-021b0b16f136\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " Oct 30 19:30:08.999726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.999706 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca\") pod \"140956c2-0f99-4931-aec6-021b0b16f136\" (UID: \"140956c2-0f99-4931-aec6-021b0b16f136\") " Oct 30 19:30:09.000014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:08.999964 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca" pod "140956c2-0f99-4931-aec6-021b0b16f136" (UID: "140956c2-0f99-4931-aec6-021b0b16f136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:09.000581 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.000410 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:09.001970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.001944 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca" pod "140956c2-0f99-4931-aec6-021b0b16f136" (UID: "140956c2-0f99-4931-aec6-021b0b16f136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:09.002813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.002793 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2" pod "140956c2-0f99-4931-aec6-021b0b16f136" (UID: "140956c2-0f99-4931-aec6-021b0b16f136"). InnerVolumeSpecName "kube-api-access-qwwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:09.101493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.101466 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwwl2\" (UniqueName: \"kubernetes.io/projected/140956c2-0f99-4931-aec6-021b0b16f136-kube-api-access-qwwl2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:09.101493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.101490 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140956c2-0f99-4931-aec6-021b0b16f136-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:09.441229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.441200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:09.767152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.767069 6514 scope.go:122] "RemoveContainer" containerID="28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744" Oct 30 19:30:09.767598 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:09.767368 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7_odh-ai-gateway-infra(78df38a3-9842-40d2-80f3-4cbe84475c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" Oct 30 19:30:09.768102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.768083 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" Oct 30 19:30:09.768176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.768116 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr" event={"ID":"140956c2-0f99-4931-aec6-021b0b16f136","Type":"ContainerDied","Data":"580b091c52f293ab30e24f69f3cdb061feae6e95022c1f276c0d80434bad317b"} Oct 30 19:30:09.768176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:09.768166 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="580b091c52f293ab30e24f69f3cdb061feae6e95022c1f276c0d80434bad317b" Oct 30 19:30:10.197665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:10.197611 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:10.330890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:10.321202 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:11.038792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:11.038763 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:11.857366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:11.857329 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:12.748930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:12.748890 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:13.624249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.624201 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:13.627532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.627503 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7171-7ee61e"] Oct 30 19:30:13.628243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.628212 6514 cpu_manager.go:414] "containerMap: removing container" podUID="140956c2-0f99-4931-aec6-021b0b16f136" containerName="revoke-keys" Oct 30 19:30:13.628319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.628244 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140956c2-0f99-4931-aec6-021b0b16f136" containerName="revoke-keys" Oct 30 19:30:13.628467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.628452 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="140956c2-0f99-4931-aec6-021b0b16f136" containerName="revoke-keys" Oct 30 19:30:13.633323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.633304 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:13.643312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.643287 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvcnk\"" Oct 30 19:30:13.643726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.643706 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7171-7ee61e"] Oct 30 19:30:13.707556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.701514 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwzt\" (UniqueName: \"kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt\") pod \"test-curl-7171-7ee61e\" (UID: \"45e70ac5-d8f7-498b-aadd-9a9981d08e2a\") " pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:13.810843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.810803 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwzt\" (UniqueName: \"kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt\") pod \"test-curl-7171-7ee61e\" (UID: \"45e70ac5-d8f7-498b-aadd-9a9981d08e2a\") " pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:13.820127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.820100 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwzt\" (UniqueName: \"kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt\") pod \"test-curl-7171-7ee61e\" (UID: \"45e70ac5-d8f7-498b-aadd-9a9981d08e2a\") " pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:13.954692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:13.954669 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:14.162301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:14.162266 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:14.236817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:14.236787 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7171-7ee61e"] Oct 30 19:30:14.236965 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:14.236946 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45e70ac5_d8f7_498b_aadd_9a9981d08e2a.slice/crio-eaac50e2d5156fb1cf2c64ae9be51e298df36140dbf38ba3339d416e778f9523 WatchSource:0}: Error finding container eaac50e2d5156fb1cf2c64ae9be51e298df36140dbf38ba3339d416e778f9523: Status 404 returned error can't find the container with id eaac50e2d5156fb1cf2c64ae9be51e298df36140dbf38ba3339d416e778f9523 Oct 30 19:30:14.623400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:14.623293 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:14.888857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:14.888781 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7171-7ee61e" event={"ID":"45e70ac5-d8f7-498b-aadd-9a9981d08e2a","Type":"ContainerStarted","Data":"eaac50e2d5156fb1cf2c64ae9be51e298df36140dbf38ba3339d416e778f9523"} Oct 30 19:30:15.308015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:15.307981 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:16.196043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:16.196002 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:16.960677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:16.960641 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:17.643548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:17.643515 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:17.919448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:17.919350 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7171-7ee61e" event={"ID":"45e70ac5-d8f7-498b-aadd-9a9981d08e2a","Type":"ContainerStarted","Data":"09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a"} Oct 30 19:30:17.936849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:17.936803 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7171-7ee61e" podStartSLOduration=2.331983028 podStartE2EDuration="4.936789827s" totalImagesPullingTime="2.604806799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:17.93322254 +0000 UTC m=+787.891041728" watchObservedRunningTime="2026-10-30 19:30:17.936789827 +0000 UTC m=+787.894609018" Oct 30 19:30:18.352688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:18.352652 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:18.687127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:18.687095 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7171-7ee61e"] Oct 30 19:30:18.691653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:18.691625 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7171-7ee61e"] Oct 30 19:30:19.053680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.053602 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.151610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.151579 6514 scope.go:122] "RemoveContainer" containerID="404349e16b9e5b6ab37911f2fda81e7e12b13ee37204dc5cd7f227f18bbfce75" Oct 30 19:30:19.153247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.153224 6514 scope.go:122] "RemoveContainer" containerID="28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744" Oct 30 19:30:19.848528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.848496 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.963700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.962337 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerStarted","Data":"fb590d592b27bd91418b0ee6ea7b39ef353f66982a4725c9569636e199f64d6b"} Oct 30 19:30:19.981372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.981331 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerStarted","Data":"d98e5f846b0fa0956e140d3dabc34ec180b6f81d52133757e0622e83d2e82d22"} Oct 30 19:30:19.982109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:19.981401 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7171-7ee61e" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" containerName="test-curl-7171-7ee61e" containerID="cri-o://09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a" gracePeriod=2 Oct 30 19:30:20.010644 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:20.010614 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7171-7ee61e\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:20.158211 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:20.158179 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7171-7ee61e\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:20.531413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.531381 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:20.882255 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:20.882167 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:30:20.993233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.993194 6514 generic.go:356] "Generic (PLEG): container finished" podID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerID="fb590d592b27bd91418b0ee6ea7b39ef353f66982a4725c9569636e199f64d6b" exitCode=60 Oct 30 19:30:20.993504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.993261 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerDied","Data":"fb590d592b27bd91418b0ee6ea7b39ef353f66982a4725c9569636e199f64d6b"} Oct 30 19:30:20.993504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.993330 6514 scope.go:122] "RemoveContainer" containerID="404349e16b9e5b6ab37911f2fda81e7e12b13ee37204dc5cd7f227f18bbfce75" Oct 30 19:30:20.993859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.993809 6514 scope.go:122] "RemoveContainer" containerID="fb590d592b27bd91418b0ee6ea7b39ef353f66982a4725c9569636e199f64d6b" Oct 30 19:30:20.994188 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:20.994138 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6_odh-ai-gateway-infra(62fc7cd4-60ad-40d6-982c-7f714d897e45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" Oct 30 19:30:20.996259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.996241 6514 generic.go:356] "Generic (PLEG): container finished" podID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerID="d98e5f846b0fa0956e140d3dabc34ec180b6f81d52133757e0622e83d2e82d22" exitCode=60 Oct 30 19:30:20.996362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.996318 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerDied","Data":"d98e5f846b0fa0956e140d3dabc34ec180b6f81d52133757e0622e83d2e82d22"} Oct 30 19:30:20.996661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.996644 6514 scope.go:122] "RemoveContainer" containerID="d98e5f846b0fa0956e140d3dabc34ec180b6f81d52133757e0622e83d2e82d22" Oct 30 19:30:20.996927 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:20.996908 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7_odh-ai-gateway-infra(78df38a3-9842-40d2-80f3-4cbe84475c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" Oct 30 19:30:20.997753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:20.997735 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6"] Oct 30 19:30:21.007885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:21.007847 6514 scope.go:122] "RemoveContainer" containerID="28268abc43a693da916aafe4200b7b47fbbdaa91d2743d04ada3884e0f238744" Oct 30 19:30:21.022142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:21.022116 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7"] Oct 30 19:30:21.159104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:21.159037 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.301927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:21.301817 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.841041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:21.841010 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.231999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.231971 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:22.235961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.235939 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:22.254037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.253409 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.373983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.366378 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd\" (UniqueName: \"kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd\") pod \"62fc7cd4-60ad-40d6-982c-7f714d897e45\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " Oct 30 19:30:22.373983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.366429 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca\") pod \"78df38a3-9842-40d2-80f3-4cbe84475c5a\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " Oct 30 19:30:22.373983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.366656 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf\" (UniqueName: \"kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf\") pod \"78df38a3-9842-40d2-80f3-4cbe84475c5a\" (UID: \"78df38a3-9842-40d2-80f3-4cbe84475c5a\") " Oct 30 19:30:22.373983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.366818 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca\") pod \"62fc7cd4-60ad-40d6-982c-7f714d897e45\" (UID: \"62fc7cd4-60ad-40d6-982c-7f714d897e45\") " Oct 30 19:30:22.377544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.375516 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca" pod "78df38a3-9842-40d2-80f3-4cbe84475c5a" (UID: "78df38a3-9842-40d2-80f3-4cbe84475c5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:22.377544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.375982 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca" pod "62fc7cd4-60ad-40d6-982c-7f714d897e45" (UID: "62fc7cd4-60ad-40d6-982c-7f714d897e45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:22.387544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.387501 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd" pod "62fc7cd4-60ad-40d6-982c-7f714d897e45" (UID: "62fc7cd4-60ad-40d6-982c-7f714d897e45"). InnerVolumeSpecName "kube-api-access-nbrxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:22.395825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.395500 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:22.398294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.398270 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf" pod "78df38a3-9842-40d2-80f3-4cbe84475c5a" (UID: "78df38a3-9842-40d2-80f3-4cbe84475c5a"). InnerVolumeSpecName "kube-api-access-w64tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:22.468815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.468791 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fc7cd4-60ad-40d6-982c-7f714d897e45-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.468815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.468814 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrxd\" (UniqueName: \"kubernetes.io/projected/62fc7cd4-60ad-40d6-982c-7f714d897e45-kube-api-access-nbrxd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.468983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.468825 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78df38a3-9842-40d2-80f3-4cbe84475c5a-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.468983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.468833 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w64tf\" (UniqueName: \"kubernetes.io/projected/78df38a3-9842-40d2-80f3-4cbe84475c5a-kube-api-access-w64tf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.569925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.569902 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt\" (UniqueName: \"kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt\") pod \"45e70ac5-d8f7-498b-aadd-9a9981d08e2a\" (UID: \"45e70ac5-d8f7-498b-aadd-9a9981d08e2a\") " Oct 30 19:30:22.571791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.571771 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt" pod "45e70ac5-d8f7-498b-aadd-9a9981d08e2a" (UID: "45e70ac5-d8f7-498b-aadd-9a9981d08e2a"). InnerVolumeSpecName "kube-api-access-8fwzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:22.671789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:22.671730 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwzt\" (UniqueName: \"kubernetes.io/projected/45e70ac5-d8f7-498b-aadd-9a9981d08e2a-kube-api-access-8fwzt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:23.019932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.019897 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" event={"ID":"62fc7cd4-60ad-40d6-982c-7f714d897e45","Type":"ContainerDied","Data":"ff6cb3fa66f6e9be9971227f3a953c04b032fa987da5db7df0161578739786d7"} Oct 30 19:30:23.019932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.019917 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6" Oct 30 19:30:23.020136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.019968 6514 scope.go:122] "RemoveContainer" containerID="fb590d592b27bd91418b0ee6ea7b39ef353f66982a4725c9569636e199f64d6b" Oct 30 19:30:23.021672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.021617 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" Oct 30 19:30:23.021672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.021627 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7" event={"ID":"78df38a3-9842-40d2-80f3-4cbe84475c5a","Type":"ContainerDied","Data":"97350229ac43b3facb454f290b13b25e4df768715f0a4da44cb1101a95c3196a"} Oct 30 19:30:23.023263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.023238 6514 generic.go:356] "Generic (PLEG): container finished" podID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" containerID="09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a" exitCode=137 Oct 30 19:30:23.023389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.023273 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:23.032767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.032750 6514 scope.go:122] "RemoveContainer" containerID="d98e5f846b0fa0956e140d3dabc34ec180b6f81d52133757e0622e83d2e82d22" Oct 30 19:30:23.042352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.042334 6514 scope.go:122] "RemoveContainer" containerID="09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a" Oct 30 19:30:23.051069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.051024 6514 scope.go:122] "RemoveContainer" containerID="09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a" Oct 30 19:30:23.051340 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:23.051322 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a\": container with ID starting with 09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a not found: ID does not exist" containerID="09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a" Oct 30 19:30:23.051396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.051350 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a"} err="failed to get container status \"09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a\": rpc error: code = NotFound desc = could not find container \"09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a\": container with ID starting with 09ee972b8c3ef88b6aab549ae0553731867337765c73a5274c574b4a4ff9239a not found: ID does not exist" Oct 30 19:30:23.052486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.052468 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6"] Oct 30 19:30:23.058380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.058356 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-01b24e-29889810-vh4f6"] Oct 30 19:30:23.079983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.079959 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7"] Oct 30 19:30:23.089207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.089186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:23.090937 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:23.090908 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7171-7ee61e\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" pod="opendatahub/test-curl-7171-7ee61e" Oct 30 19:30:23.091465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.091447 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889810-h6sn7"] Oct 30 19:30:23.924761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:23.924728 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.159807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:24.159775 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" path="/var/lib/kubelet/pods/45e70ac5-d8f7-498b-aadd-9a9981d08e2a/volumes" Oct 30 19:30:24.160325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:24.160310 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" path="/var/lib/kubelet/pods/62fc7cd4-60ad-40d6-982c-7f714d897e45/volumes" Oct 30 19:30:24.160681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:24.160668 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" path="/var/lib/kubelet/pods/78df38a3-9842-40d2-80f3-4cbe84475c5a/volumes" Oct 30 19:30:24.835914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:24.835848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.630380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:25.630353 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:26.193964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.193933 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7168-99758b"] Oct 30 19:30:26.194557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194544 6514 cpu_manager.go:414] "containerMap: removing container" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194559 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194566 6514 cpu_manager.go:414] "containerMap: removing container" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.194608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194571 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.194608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194600 6514 cpu_manager.go:414] "containerMap: removing container" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194605 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194626 6514 cpu_manager.go:414] "containerMap: removing container" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194630 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194644 6514 cpu_manager.go:414] "containerMap: removing container" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" containerName="test-curl-7171-7ee61e" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194648 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" containerName="test-curl-7171-7ee61e" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194656 6514 cpu_manager.go:414] "containerMap: removing container" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194660 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194747 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194755 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.194998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194763 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194771 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e70ac5-d8f7-498b-aadd-9a9981d08e2a" containerName="test-curl-7171-7ee61e" Oct 30 19:30:26.194998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194777 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.194998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194782 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fc7cd4-60ad-40d6-982c-7f714d897e45" containerName="cleanup" Oct 30 19:30:26.194998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.194787 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.198308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.198288 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:26.206916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.206814 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvcnk\"" Oct 30 19:30:26.225804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.225775 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7168-99758b"] Oct 30 19:30:26.230263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.230231 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxzx\" (UniqueName: \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\") pod \"test-curl-7168-99758b\" (UID: \"8cfe5572-6003-49ad-937d-b833210749ff\") " pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:26.331237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.331202 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxzx\" (UniqueName: \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\") pod \"test-curl-7168-99758b\" (UID: \"8cfe5572-6003-49ad-937d-b833210749ff\") " pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:26.341244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.341220 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxzx\" (UniqueName: \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\") pod \"test-curl-7168-99758b\" (UID: \"8cfe5572-6003-49ad-937d-b833210749ff\") " pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:26.526552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.526477 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:26.562410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.562379 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:26.891376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.891347 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7168-99758b"] Oct 30 19:30:26.892325 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:26.892288 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cfe5572_6003_49ad_937d_b833210749ff.slice/crio-b1cb0a3b7f5d88c3279957dcdbdfff7a7b1b4a00166c7b12d63ce9a9b435e6e5 WatchSource:0}: Error finding container b1cb0a3b7f5d88c3279957dcdbdfff7a7b1b4a00166c7b12d63ce9a9b435e6e5: Status 404 returned error can't find the container with id b1cb0a3b7f5d88c3279957dcdbdfff7a7b1b4a00166c7b12d63ce9a9b435e6e5 Oct 30 19:30:26.904965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.904941 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j"] Oct 30 19:30:26.905714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.905694 6514 cpu_manager.go:414] "containerMap: removing container" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.905714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.905716 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78df38a3-9842-40d2-80f3-4cbe84475c5a" containerName="cleanup" Oct 30 19:30:26.920584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.920561 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.922838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.922813 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j"] Oct 30 19:30:26.943690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.943667 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.943899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.943838 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.944067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.944043 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.944266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.944251 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.945213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.945182 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.945307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.945228 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.945366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.945319 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp8j\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.945366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.945352 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:26.945487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:26.945377 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.040484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.040460 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz"] Oct 30 19:30:27.045287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.045270 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.046288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046261 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046351 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046383 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046453 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp8j\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046476 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046502 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046627 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046666 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046760 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.046911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.046785 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.047180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.047162 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.047354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.047338 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.047480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.047459 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.047548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.047463 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.048653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.048631 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.048723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.048679 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.059200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.058448 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz"] Oct 30 19:30:27.060998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.060971 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.060998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.060988 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp8j\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.122681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.122644 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7168-99758b" event={"ID":"8cfe5572-6003-49ad-937d-b833210749ff","Type":"ContainerStarted","Data":"2a679e611eacf2dc7fb1cdfcc6f2f6b1806007a60b70b9d205c0fdaba022215c"} Oct 30 19:30:27.122827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.122693 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7168-99758b" event={"ID":"8cfe5572-6003-49ad-937d-b833210749ff","Type":"ContainerStarted","Data":"b1cb0a3b7f5d88c3279957dcdbdfff7a7b1b4a00166c7b12d63ce9a9b435e6e5"} Oct 30 19:30:27.140088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.140006 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7168-99758b" podStartSLOduration=1.139989121 podStartE2EDuration="1.139989121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:27.137315368 +0000 UTC m=+797.095134558" watchObservedRunningTime="2026-10-30 19:30:27.139989121 +0000 UTC m=+797.097808310" Oct 30 19:30:27.147816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.147794 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.147903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.147828 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb47r\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.147903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.147855 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.148057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.148029 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.148170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.148097 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.148233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.148167 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.148267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.148260 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.148297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.148279 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.148326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.148296 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249110 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249156 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249293 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249414 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249450 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249477 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249538 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249736 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249778 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb47r\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.249850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.249818 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.250633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.250604 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.250760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.250717 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.250989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.250962 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.251240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.251217 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.252184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.252161 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.252764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.252738 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.259124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.259100 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl"] Oct 30 19:30:27.266845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.266825 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:27.267069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.267046 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb47r\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.269422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.269403 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.277423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.276893 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.300989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.293712 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl"] Oct 30 19:30:27.355141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.355097 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.356455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.356433 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.357064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.357044 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.357266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.357247 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.357368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.357352 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.357472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.357457 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.357704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.357687 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.358056 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.358035 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d786x\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.358347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.358331 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.362407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.362350 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.416327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.416275 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:27.460553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.460514 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.461166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.461141 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.461458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.461441 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.461598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.461582 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.461717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.461702 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d786x\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.461843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.461830 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.462068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.462050 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.462400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.462385 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.462513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.462500 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.463474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.463450 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.464307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.464283 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.464700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.464680 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.482849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.482820 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.492037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.483265 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.506012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.505893 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d786x\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.506845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.506650 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.507353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.507325 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.568887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.567089 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.665062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.658855 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:27.764352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.764283 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm"] Oct 30 19:30:27.823242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.823212 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:27.903829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.903788 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm"] Oct 30 19:30:27.910007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.909977 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j"] Oct 30 19:30:27.986215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:27.986180 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.001159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.001129 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.001462 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.001365 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.001891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.001847 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.002112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.002092 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.002333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.002314 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmfr\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.002579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.002562 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.002811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.002709 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.003036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.003000 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.058217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.054403 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.058217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.054708 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.058217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.055002 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200410 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200516 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200543 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmfr\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200678 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200740 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200812 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200849 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200910 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.200991 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.216564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.203115 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.257085 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.238091 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.257085 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.238379 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.257085 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.238959 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.257085 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.239583 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.670891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.655382 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.670891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.656084 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmfr\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.720888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.671594 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.720888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.674172 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.755896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.739361 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:28.798920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.785070 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz"] Oct 30 19:30:28.848430 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.848335 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.931890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.921978 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.931890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:28.922074 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:29.125891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.113178 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" event={"ID":"40bebcec-3413-4a53-ad7c-5c73a996bcb3","Type":"ContainerStarted","Data":"259ca271c3ea152d893a3b1d06be4f77487d3387fe2dc6a1c9e9e6b995754aab"} Oct 30 19:30:29.281216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.280086 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:29.623883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.622386 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:29.678901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.653609 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:29.752382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.749970 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:29.752382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.750857 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:29.963099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.908718 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:30.000898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:29.996566 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:30.077360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.057321 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.134577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.125069 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.134577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.126194 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:30.654140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.653960 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:30.702930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.654336 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:30.706885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.678945 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.735439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.688225 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:30.831289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.831247 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:30.831482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.831463 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:30.862348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.755428 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:30:31.430059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.416369 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl"] Oct 30 19:30:31.430059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.416559 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:30:31.430059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.298997 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.430059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.333050 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:30:31.430059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.333141 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.430059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:30.790195 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:30:31.461302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.333084 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:31.461302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.333110 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 19:30:31.545564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.545338 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:30:31.545564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.545434 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:30:31.545564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.545481 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:30:31.545564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.545513 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:31.633590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.633555 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.794943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.716090 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" event={"ID":"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663","Type":"ContainerStarted","Data":"8cc366c0b2ea260115f004a850f50dc47fe59c39f1201f8cce2c1c7ada62bc11"} Oct 30 19:30:31.865886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.777171 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.865886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.777204 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:31.865886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.777227 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.865886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.828571 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:30:31.865886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:31.848651 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" podUID="2c01c154-6ca2-4dcd-b558-2a3790449664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.425149 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.439049 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.454725 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.455091 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.497937 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.498382 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:32.539943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.498617 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:32.907982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:32.864808 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm"] Oct 30 19:30:33.225954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.225579 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:30:33.225954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.204505 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:33.234894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.204593 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:33.235116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.204547 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:33.504074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.490043 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:33.504074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.490521 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:33.504074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.490896 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:30:33.807326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.766311 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" event={"ID":"f3b2a985-fc38-41db-a6c3-0352430f1595","Type":"ContainerStarted","Data":"77cec4f25e9cad2cbdd9db12c41800c80b6f16ca0a9f6b0973e9b8b409e50ca5"} Oct 30 19:30:33.807326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:33.781429 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" event={"ID":"40bebcec-3413-4a53-ad7c-5c73a996bcb3","Type":"ContainerStarted","Data":"46de497fb81788d68f1914f9ec4a57a5f39b9e5a01d25e17a719cf1b8101c49e"} Oct 30 19:30:34.112939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.092730 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.246010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.239330 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.257108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.256984 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.258007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.257892 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:34.435606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.435503 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:34.471467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.471419 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:34.521636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.513965 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.543373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.522390 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:34.543373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.522731 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.577034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.557953 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.580884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.580158 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podStartSLOduration=8.580134902 podStartE2EDuration="8.580134902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:33.943204162 +0000 UTC m=+803.901023350" watchObservedRunningTime="2026-10-30 19:30:34.580134902 +0000 UTC m=+804.537954092" Oct 30 19:30:34.663201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.661471 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:34.669959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.669931 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:34.670151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.670127 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" event={"ID":"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663","Type":"ContainerStarted","Data":"a7c532649e420f8ff741c515543285bd8b1863e2739eb9922aeff25052d5e9d2"} Oct 30 19:30:34.670270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.670255 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" event={"ID":"85766866-9614-45d4-a5b3-fba1fcae567d","Type":"ContainerStarted","Data":"f94d7dede5e5b3be1c6d4e80f4eb3a6f701e82733798bf92c1576c51e12ede83"} Oct 30 19:30:34.670593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.670429 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:34.798546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.797157 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podStartSLOduration=7.797137501 podStartE2EDuration="7.797137501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:34.717790992 +0000 UTC m=+804.675610180" watchObservedRunningTime="2026-10-30 19:30:34.797137501 +0000 UTC m=+804.754956689" Oct 30 19:30:34.898094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.889890 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:30:34.903110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:34.903087 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:30:35.015691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.013193 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:30:35.069890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.067811 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:30:35.132959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.129911 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7168-99758b"] Oct 30 19:30:35.132959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.130117 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7168-99758b" podUID="8cfe5572-6003-49ad-937d-b833210749ff" containerName="test-curl-7168-99758b" containerID="cri-o://2a679e611eacf2dc7fb1cdfcc6f2f6b1806007a60b70b9d205c0fdaba022215c" gracePeriod=2 Oct 30 19:30:35.134854 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:35.134239 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:35.174478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.174448 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:30:35.233156 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:35.228410 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:35.282934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.272632 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7168-99758b"] Oct 30 19:30:35.290799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.290722 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:35.290799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.290776 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:35.418533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.418499 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:35.481897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.474692 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:35.481897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.474742 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:35.674714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.674627 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" event={"ID":"85766866-9614-45d4-a5b3-fba1fcae567d","Type":"ContainerStarted","Data":"c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246"} Oct 30 19:30:35.676844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.676818 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" event={"ID":"f3b2a985-fc38-41db-a6c3-0352430f1595","Type":"ContainerStarted","Data":"96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438"} Oct 30 19:30:35.725526 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:35.725483 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:35.757883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.753135 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:35.762497 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:35.757027 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:35.763126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.763062 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podStartSLOduration=8.763043734 podStartE2EDuration="8.763043734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:35.720226356 +0000 UTC m=+805.678045546" watchObservedRunningTime="2026-10-30 19:30:35.763043734 +0000 UTC m=+805.720862924" Oct 30 19:30:35.790445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.790389 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podStartSLOduration=8.790371135000001 podStartE2EDuration="8.790371135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:35.7873323 +0000 UTC m=+805.745151505" watchObservedRunningTime="2026-10-30 19:30:35.790371135 +0000 UTC m=+805.748190324" Oct 30 19:30:35.814889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:35.800339 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:36.352891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:36.352814 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.353381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:36.352911 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:36.445166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:36.445111 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.445512 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:36.445456 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:37.026473 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.026433 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.038896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.038838 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:37.039445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.026786 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.039678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.039650 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:37.047616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.047531 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.107205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.107156 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b"] Oct 30 19:30:37.135661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.135624 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8cfe5572-6003-49ad-937d-b833210749ff" containerName="test-curl-7168-99758b" Oct 30 19:30:37.135923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.135899 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfe5572-6003-49ad-937d-b833210749ff" containerName="test-curl-7168-99758b" Oct 30 19:30:37.136635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.136615 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfe5572-6003-49ad-937d-b833210749ff" containerName="test-curl-7168-99758b" Oct 30 19:30:37.151051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.148580 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b"] Oct 30 19:30:37.151051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.148649 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:37.151051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.149100 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.186888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.158260 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2ae6628f-openshift-default-dockercfg-2cqb9\"" Oct 30 19:30:37.270060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.270013 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.287194 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.287096 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.303660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.303623 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.304285 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.304261 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.304523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.304505 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.304734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.304719 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.304896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.304858 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxmx\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.305057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.305038 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.305248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.305231 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.305595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.305573 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.395949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.387686 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.395949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.392732 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.395949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.392778 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:37.395949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.394491 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.415155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.410750 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.446091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.446045 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.462900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.462837 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.479803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.479768 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.480100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.480069 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:37.524851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.523664 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.550608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.547515 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.550608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.547591 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.550608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.547636 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.550608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.547769 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.550608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.547964 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.554881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.548013 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxmx\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.554881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.553100 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.554881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.553194 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.554881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.553362 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.559992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.559962 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.568135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.560925 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.577653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.561425 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.577798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.562066 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.577926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.562337 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.578834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.578805 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.609460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.609426 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxmx\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.610103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.610051 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.644743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.644654 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.702269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.695629 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo\") pod \"e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.705102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.705068 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.705297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.705271 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:37.732992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.732688 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:37.748539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.748509 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.748758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:37.748717 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:38.093885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.090971 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:38.203076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.202305 6514 generic.go:356] "Generic (PLEG): container finished" podID="8cfe5572-6003-49ad-937d-b833210749ff" containerID="2a679e611eacf2dc7fb1cdfcc6f2f6b1806007a60b70b9d205c0fdaba022215c" exitCode=137 Oct 30 19:30:38.234515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.234477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:38.268136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.267649 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:38.268136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.267715 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:38.397415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.394982 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:38.397415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.395567 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:30:38.454908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.445769 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:38.454908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.446089 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:38.481786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.463565 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:38.481786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.463969 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:38.576921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.572580 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:38.630889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.613429 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:30:38.632596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.632562 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:30:38.633131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.633109 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:30:38.738742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.736794 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:38.738742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.736855 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:38.798446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.796258 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:38.798446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.796327 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:38.907935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:38.890482 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:39.117578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.117545 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:39.295666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.295560 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:39.317884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.296780 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:39.338234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.319003 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:39.353093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.351119 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:39.426104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.426060 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:39.493258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.482516 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:39.493957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.493922 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:39.647995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.634114 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:39.778889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.712401 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:40.155522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.772751 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.155805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.155773 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:40.180978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.022157 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:40.181183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.084002 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1cb0a3b7f5d88c3279957dcdbdfff7a7b1b4a00166c7b12d63ce9a9b435e6e5" Oct 30 19:30:40.213888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.006268 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:40.213888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.187989 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:40.213888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.188139 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:40.213888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:39.918220 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.213888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.188763 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:40.353222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.300466 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:40.353222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.300543 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:40.502243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.501949 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:40.625940 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:40.621808 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:40.656963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.629618 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.656963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.637709 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:40.656963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.632947 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:40.657612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.657538 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:40.660228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.660208 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:30:40.660422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.660408 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b"] Oct 30 19:30:40.662978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.662956 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.663174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.663149 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:40.756886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.755076 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.756886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.755240 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:40.761167 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.761134 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.761303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.761184 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:40.773977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.773939 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\" (UniqueName: \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\") pod \"8cfe5572-6003-49ad-937d-b833210749ff\" (UID: \"8cfe5572-6003-49ad-937d-b833210749ff\") " Oct 30 19:30:40.983680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.983640 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx" pod "8cfe5572-6003-49ad-937d-b833210749ff" (UID: "8cfe5572-6003-49ad-937d-b833210749ff"). InnerVolumeSpecName "kube-api-access-zrxzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:40.984919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.984832 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:40.985583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.985565 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\" (UniqueName: \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\") pod \"8cfe5572-6003-49ad-937d-b833210749ff\" (UID: \"8cfe5572-6003-49ad-937d-b833210749ff\") " Oct 30 19:30:40.986019 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:40.985874 6514 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8cfe5572-6003-49ad-937d-b833210749ff/volumes/kubernetes.io~projected/kube-api-access-zrxzx Oct 30 19:30:40.986176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.986160 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx" pod "8cfe5572-6003-49ad-937d-b833210749ff" (UID: "8cfe5572-6003-49ad-937d-b833210749ff"). InnerVolumeSpecName "kube-api-access-zrxzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:40.988707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:40.988686 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:30:41.063890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.030914 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxzx\" (UniqueName: \"kubernetes.io/projected/8cfe5572-6003-49ad-937d-b833210749ff-kube-api-access-zrxzx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:41.175148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.169879 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:41.253934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.176196 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:41.363964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.322528 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:41.471889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.423083 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:41.471889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.423154 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:41.496891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.495397 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz"] Oct 30 19:30:41.637483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.637219 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:41.666075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.664566 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:30:41.672828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.672804 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz"] Oct 30 19:30:41.821017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.820980 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" event={"ID":"766172a6-4e89-4544-b021-6ea59c0c7789","Type":"ContainerStarted","Data":"b3f0582fbea461514c3d3a58962a196904f5b04489b7cb9e77d1595e904c94af"} Oct 30 19:30:41.821286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.690901 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:41.821484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.821456 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:41.821601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.691148 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:41.822298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.690972 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:41.832857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.715428 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:41.832857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.832745 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:41.832857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.777224 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:41.832857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.832816 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:41.833178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.715076 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:41.834636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.784982 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:41.840638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.840621 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:41.853896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.690164 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:41.854130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.854093 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:30:41.867892 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:41.864340 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:41.914367 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.874150 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-77930c58\"" Oct 30 19:30:41.914367 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.874278 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-77930c58-dockercfg-qjkpb\"" Oct 30 19:30:41.985889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:41.985077 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk"] Oct 30 19:30:42.120933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.120276 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npw5\" (UniqueName: \"kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5\") pod \"payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.120933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.120607 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.191894 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:42.186854 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:42.215432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.215388 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:42.282540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.282500 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.283089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.283062 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npw5\" (UniqueName: \"kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5\") pod \"payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.284749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.284726 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.286425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.286402 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:42.286600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.286575 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:42.329900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.305077 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk"] Oct 30 19:30:42.329900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.326615 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:42.329900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.327236 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:42.384992 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:42.381518 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:42.462551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.462516 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:42.462797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.462766 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:42.585799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.585764 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npw5\" (UniqueName: \"kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5\") pod \"payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.615154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.615115 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwlc\" (UniqueName: \"kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc\") pod \"payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:42.615483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.615460 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:42.646914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.646783 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:42.665785 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:42.648592 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-99758b\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="8cfe5572-6003-49ad-937d-b833210749ff" pod="opendatahub/test-curl-7168-99758b" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.686388 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfe5572-6003-49ad-937d-b833210749ff" path="/var/lib/kubelet/pods/8cfe5572-6003-49ad-937d-b833210749ff/volumes" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.689310 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.689355 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.689387 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.689437 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.689469 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.689485 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.688188 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.690562 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:42.703095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.688471 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:42.731970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.716917 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwlc\" (UniqueName: \"kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc\") pod \"payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:42.731970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.717023 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:42.731970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.717833 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:30:42.731970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.718162 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:42.767246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.767211 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:42.822039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.821995 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:42.833011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.797472 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:30:42.982084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:42.951187 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:43.121908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.120556 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwlc\" (UniqueName: \"kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc\") pod \"payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:43.170896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.165224 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" event={"ID":"766172a6-4e89-4544-b021-6ea59c0c7789","Type":"ContainerStarted","Data":"2aa349aa90dc52294905a76baa76378ab4ab70a58d980bd88a52431a23a83bb3"} Oct 30 19:30:43.282669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.282591 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:30:43.321902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.284642 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:43.321902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.284694 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:43.402048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.401999 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:43.412536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.412502 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:43.825309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.825238 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:43.827918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.827892 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:43.832815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.832765 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:30:43.834808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.834782 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:30:43.837157 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.837138 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:43.884766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.837942 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:43.917954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.905129 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:43.917954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.905190 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:43.971886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.958122 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:43.971886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.958199 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:44.021546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:43.984046 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:44.080139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.080102 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:44.183584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.125779 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:30:44.230927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.216959 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" podStartSLOduration=8.216940918 podStartE2EDuration="8.216940918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:43.222538476 +0000 UTC m=+813.180357665" watchObservedRunningTime="2026-10-30 19:30:44.216940918 +0000 UTC m=+814.174760107" Oct 30 19:30:44.261045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.259511 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:30:44.306834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.306808 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:44.319447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.319423 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:30:44.544058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.541390 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz"] Oct 30 19:30:44.611474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.611440 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:44.630797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.630771 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:30:44.663528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.663497 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk"] Oct 30 19:30:44.666890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.666846 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:30:44.666971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.666908 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 19:30:44.746955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.744339 6514 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:30:44.746955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.744389 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 19:30:44.902532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:44.902497 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:45.314651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.314608 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" event={"ID":"bdde9ebb-2a7a-4cca-95f3-d390b8947867","Type":"ContainerStarted","Data":"4b0c64c6d6581327290e35df8dc0545f02cc150090c35073a87fd770945f701a"} Oct 30 19:30:45.314842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.314728 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:45.314842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.314762 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" event={"ID":"bdde9ebb-2a7a-4cca-95f3-d390b8947867","Type":"ContainerStarted","Data":"33e6d974c4520f0696356c22f44ba8d31b6d725f347e74c5ec8ac51ffb632dfa"} Oct 30 19:30:45.318307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.318274 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" event={"ID":"e8156308-0fbb-4b06-8b85-2a6bee8729f4","Type":"ContainerStarted","Data":"eedea8b85d9ac5afba9f694da6cf568c98ce65573995b5a82973a157229d8bc2"} Oct 30 19:30:45.318681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.318661 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:45.318754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.318702 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" event={"ID":"e8156308-0fbb-4b06-8b85-2a6bee8729f4","Type":"ContainerStarted","Data":"43fe0ab562da6cda995b4f995174a4ebc82f3695ea51b12cc29869c35e33646a"} Oct 30 19:30:45.336055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.336007 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" podStartSLOduration=4.335991355 podStartE2EDuration="4.335991355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:45.333235009 +0000 UTC m=+815.291054228" watchObservedRunningTime="2026-10-30 19:30:45.335991355 +0000 UTC m=+815.293810546" Oct 30 19:30:45.353615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.353576 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" podStartSLOduration=4.353563549 podStartE2EDuration="4.353563549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:45.352944272 +0000 UTC m=+815.310763463" watchObservedRunningTime="2026-10-30 19:30:45.353563549 +0000 UTC m=+815.311382737" Oct 30 19:30:45.824336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.824258 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:45.939808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.937717 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:45.947988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:45.946661 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:30:46.378038 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.378008 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:46.379427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.379402 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:30:46.445455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.445424 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:46.627180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.627136 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:46.638939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.638799 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw"] Oct 30 19:30:46.645355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.645328 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw"] Oct 30 19:30:46.645491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.645461 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:46.651158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.651135 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2ae6628f\"" Oct 30 19:30:46.785892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.781130 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:46.802888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.788672 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:46.802888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.788903 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fkd\" (UniqueName: \"kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd\") pod \"payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:46.898933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.891912 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:46.898933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.892104 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fkd\" (UniqueName: \"kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd\") pod \"payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:46.898933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.893295 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:46.935886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:46.928362 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fkd\" (UniqueName: \"kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd\") pod \"payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:47.012890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:47.011601 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:47.399368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:47.399339 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw"] Oct 30 19:30:47.400224 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:47.400188 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1380c68e_9976_4720_ac62_a167f0e5dc60.slice/crio-3eb5a6f986ae218ca305e70539912c36dfb247c5b29c5c86d06708242d4527c2 WatchSource:0}: Error finding container 3eb5a6f986ae218ca305e70539912c36dfb247c5b29c5c86d06708242d4527c2: Status 404 returned error can't find the container with id 3eb5a6f986ae218ca305e70539912c36dfb247c5b29c5c86d06708242d4527c2 Oct 30 19:30:47.754974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:47.754945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:48.347244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:48.347207 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" event={"ID":"1380c68e-9976-4720-ac62-a167f0e5dc60","Type":"ContainerStarted","Data":"9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71"} Oct 30 19:30:48.347708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:48.347272 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" event={"ID":"1380c68e-9976-4720-ac62-a167f0e5dc60","Type":"ContainerStarted","Data":"3eb5a6f986ae218ca305e70539912c36dfb247c5b29c5c86d06708242d4527c2"} Oct 30 19:30:48.347708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:48.347372 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:30:48.388963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:48.388912 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" podStartSLOduration=2.388897025 podStartE2EDuration="2.388897025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:48.38440313 +0000 UTC m=+818.342222318" watchObservedRunningTime="2026-10-30 19:30:48.388897025 +0000 UTC m=+818.346716212" Oct 30 19:30:48.808030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:48.808000 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:49.556780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:49.556744 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:50.345415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:50.345384 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:50.469420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:50.469383 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:50.615619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:50.615539 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:50.751506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:50.751464 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:51.035982 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:51.035849 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:30:51.538039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:51.538001 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:52.353104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:52.353020 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:53.373930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:53.369852 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.077816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.077782 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl"] Oct 30 19:30:54.087097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.087069 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.094168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.094139 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2ae6628f-serving-cert\"" Oct 30 19:30:54.099883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.099827 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl"] Oct 30 19:30:54.160007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.159977 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.225064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.225021 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.229072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.229036 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62cmh\" (UniqueName: \"kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.246898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.229371 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.247326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.247293 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.279941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.279888 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.353573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.353486 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.353573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.353542 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62cmh\" (UniqueName: \"kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.353753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.353589 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.353753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.353656 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.353975 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:54.353949 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-serving-cert: secret "maas-api-e2e-ait-2ae6628f-serving-cert" not found Oct 30 19:30:54.354224 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:30:54.354209 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls podName:bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:54.854185715 +0000 UTC m=+824.812004905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls") pod "maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" (UID: "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9") : secret "maas-api-e2e-ait-2ae6628f-serving-cert" not found Oct 30 19:30:54.355003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.354973 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.357884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.357848 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.365564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.365539 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62cmh\" (UniqueName: \"kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.633565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.633478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.876888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.873392 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:54.889995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:54.883624 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\") pod \"maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:55.011976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:55.011929 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:55.333600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:55.333568 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:55.334915 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:30:55.334887 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf6f95f_0e6c_445d_b40e_b2e6d0ffc7f9.slice/crio-9d6c53f260893f381b6609c53c81b5128a9b30145dc1a658f97ab6a945edc8c9 WatchSource:0}: Error finding container 9d6c53f260893f381b6609c53c81b5128a9b30145dc1a658f97ab6a945edc8c9: Status 404 returned error can't find the container with id 9d6c53f260893f381b6609c53c81b5128a9b30145dc1a658f97ab6a945edc8c9 Oct 30 19:30:55.335087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:55.335054 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl"] Oct 30 19:30:55.336729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:55.336708 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:30:55.472912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:55.470299 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" event={"ID":"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9","Type":"ContainerStarted","Data":"9d6c53f260893f381b6609c53c81b5128a9b30145dc1a658f97ab6a945edc8c9"} Oct 30 19:30:56.331824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:56.331796 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:30:56.350891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:56.350841 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:56.507615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:56.503369 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" event={"ID":"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9","Type":"ContainerStarted","Data":"f56332301430a6599bc7a7768108fc2d475dbea54625997980b00abf6bdace93"} Oct 30 19:30:56.507615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:56.504450 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:30:56.545148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:56.545087 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" podStartSLOduration=2.129792548 podStartE2EDuration="2.545068024s" totalImagesPullingTime="415.275476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:56.540746152 +0000 UTC m=+826.498565341" watchObservedRunningTime="2026-10-30 19:30:56.545068024 +0000 UTC m=+826.502887214" Oct 30 19:30:57.233771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:57.233741 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:57.338111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:57.338088 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:30:57.372082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:57.372053 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:57.647662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:57.647331 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.098099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:58.098067 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.358565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:58.338068 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.462941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:58.460408 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:59.339271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:59.339240 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:59.397120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:30:59.397091 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:31:00.048470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:00.048440 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:01.189758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:01.189721 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:01.659688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:01.659614 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:01.935442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:01.935410 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.073059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:02.073028 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.217873 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:02.217747 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.336831 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:02.336800 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.674548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:02.674472 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:31:03.055018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:03.054974 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:03.252339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:03.252309 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:31:03.935611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:03.935566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:04.384929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:04.384831 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:04.922883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:04.918905 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28"] Oct 30 19:31:04.943308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:04.937200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:04.959240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:04.946290 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.010929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.009626 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-229db90d-openshift-default-dockercfg-tl8ll\"" Oct 30 19:31:05.087902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.064689 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28"] Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089447 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089521 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bgg\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089666 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089740 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089829 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089882 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089925 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.089965 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.103991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.090007 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.192998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.192959 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.212806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.212773 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bgg\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.213235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.213209 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.213446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.213427 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.213653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.213634 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.213802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.213785 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.213962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.213941 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.214131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.214111 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.214271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.214256 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.226565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.220208 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.226737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.212663 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.226848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.220995 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.227015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.222384 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.227169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.222618 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.240134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.240108 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.297885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.294470 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.326340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.326291 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:05.411895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.410515 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.429892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.422253 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.519322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.499511 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.559825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.559700 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bgg\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg\") pod \"e2e-ait-229db90d-openshift-default-8696475765-tkr28\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.713894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.621774 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:05.714458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.637950 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.779971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.770376 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.835285 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:05.835233 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.010676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.010632 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:31:06.330677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.328738 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:06.330677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.329237 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:31:06.389505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.386988 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.389505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.387526 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.431227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.431185 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:06.432487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.432462 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:31:06.522907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.515952 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.522907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.516527 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:31:06.722014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.721963 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.723370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.723336 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:06.724090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.724067 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:31:06.939889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.815110 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.180891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:06.827337 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:07.190900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.018618 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:07.190900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.120005 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:07.190900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.185475 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.190900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.180449 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:07.190900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.186007 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:31:07.190900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.186058 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:31:07.565076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.467972 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.565076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.369949 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:07.565076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.513967 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:31:07.565076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.538958 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.565076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:07.539796 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:08.104007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.015509 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:08.328893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.314683 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz" podUID="b4552499-23ca-48dd-9210-1e79e6ad04e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded" Oct 30 19:31:08.641536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.326970 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:08.687992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.537183 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:08.688334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.539431 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.918118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.918019 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:08.943543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.943499 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:08.996815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:08.996770 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.050133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.019705 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:09.050133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.019797 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:09.050133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.019839 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:09.072051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.072004 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tlbtd" podUID="2c01c154-6ca2-4dcd-b558-2a3790449664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.152019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.151966 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.703079 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:31:09.621494 6514 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.021s" Oct 30 19:31:09.770503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.770478 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28"] Oct 30 19:31:09.804996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.703241 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:09.815599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.703261 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:09.818730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.703196 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.884892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.842395 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" event={"ID":"591aa71d-e36f-42ee-8299-1569c5737b08","Type":"ContainerStarted","Data":"a8f83d7ce822245a00278a60e037d955183027e51058e72e6ab5c7412b10623d"} Oct 30 19:31:09.933001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.932958 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:09.974045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.973955 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:09.974278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:09.974249 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.002089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.002048 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.004360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.004049 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:10.004635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.004560 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.012897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.010511 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.047031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.046991 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:10.047333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.047288 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.055133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.049447 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:10.055133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.049490 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.086586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.086464 6514 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:10.086586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.086521 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.201926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.164952 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:10.548773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.548739 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:10.771915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.764951 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:10.816887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.814012 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:10.816887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.815891 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:10.841952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.833452 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.935185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.934048 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.950383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:10.948688 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.022888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.013056 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.022888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.014618 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:11.086265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.086227 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" event={"ID":"591aa71d-e36f-42ee-8299-1569c5737b08","Type":"ContainerStarted","Data":"dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab"} Oct 30 19:31:11.169889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.166342 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" podStartSLOduration=7.166323194 podStartE2EDuration="7.166323194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:11.110337938 +0000 UTC m=+841.068157126" watchObservedRunningTime="2026-10-30 19:31:11.166323194 +0000 UTC m=+841.124142382" Oct 30 19:31:11.169889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.169368 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.190951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.188757 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:11.667951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.663848 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:11.736888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.721290 6514 patch_prober.go:29] interesting pod/e2e-ait-229db90d-openshift-default-8696475765-tkr28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:31:11.736888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.721368 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:31:11.817019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.816971 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:11.817961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.817927 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:11.840003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.839968 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:11.840303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.840274 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.859885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.858570 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.952920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:11.951816 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.016886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.016056 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:12.031894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.017995 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.086889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.085973 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:12.086889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.086045 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.171836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.169513 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.309286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.306952 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:12.440485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.440397 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:12.452218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.451726 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:12.452218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.451787 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.655953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.655085 6514 patch_prober.go:29] interesting pod/e2e-ait-229db90d-openshift-default-8696475765-tkr28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 19:31:12.655953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.655156 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 19:31:12.830894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.830579 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.855557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.855502 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.879775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.876561 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.907651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.907567 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:12.967054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:12.959120 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.017537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.017495 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.018614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.018582 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.187342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.187283 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.648355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.648266 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:13.836887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.835907 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.880608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.880569 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.881128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.880635 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:13.976889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:13.971384 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:14.008354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.008326 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:31:14.019904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.017845 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:14.026760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.026725 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:14.128400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.127203 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:31:14.130583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.130562 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:31:14.132301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.132283 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:31:14.141823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.140704 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:31:14.142365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.142345 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:14.221610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.221574 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:14.241034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.240793 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:14.244366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.244342 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:31:14.247930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.247907 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:31:14.267448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.267241 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:31:14.284335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.284286 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:14.872705 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:14.872675 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.075707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.075670 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.300629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.300586 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.515841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.513855 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.614123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.614053 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb"] Oct 30 19:31:15.620028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.620009 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.629136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.629113 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb"] Oct 30 19:31:15.629551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.629532 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-229db90d\"" Oct 30 19:31:15.632878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.632845 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-229db90d-dockercfg-pcxp2\"" Oct 30 19:31:15.666759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.666733 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.767095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.767061 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.767255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.767114 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965sc\" (UniqueName: \"kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc\") pod \"payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.875936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.875065 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.875936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.875148 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965sc\" (UniqueName: \"kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc\") pod \"payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.886897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.876621 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.904911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.902816 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965sc\" (UniqueName: \"kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc\") pod \"payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:15.949913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:15.945878 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:16.414169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:16.414144 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb"] Oct 30 19:31:16.414649 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:31:16.414626 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10aa3ab8_028e_4b43_a283_e2082f5c7a6c.slice/crio-732fac892b83cb26bdc19a279a99ff3bef944b15657a76b6469ec8e2dfcdd809 WatchSource:0}: Error finding container 732fac892b83cb26bdc19a279a99ff3bef944b15657a76b6469ec8e2dfcdd809: Status 404 returned error can't find the container with id 732fac892b83cb26bdc19a279a99ff3bef944b15657a76b6469ec8e2dfcdd809 Oct 30 19:31:16.465214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:16.465173 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:17.247357 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:17.247298 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:17.332857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:17.330181 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" event={"ID":"10aa3ab8-028e-4b43-a283-e2082f5c7a6c","Type":"ContainerStarted","Data":"70ee22904ae5c02a39ded24114ba0130860dd622e237cd4ea1c521582bc2bc2c"} Oct 30 19:31:17.333060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:17.332917 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:17.333060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:17.332953 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" event={"ID":"10aa3ab8-028e-4b43-a283-e2082f5c7a6c","Type":"ContainerStarted","Data":"732fac892b83cb26bdc19a279a99ff3bef944b15657a76b6469ec8e2dfcdd809"} Oct 30 19:31:17.381170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:17.381116 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" podStartSLOduration=2.381099658 podStartE2EDuration="2.381099658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:17.374077773 +0000 UTC m=+847.331896976" watchObservedRunningTime="2026-10-30 19:31:17.381099658 +0000 UTC m=+847.338918846" Oct 30 19:31:18.241128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:18.241100 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.162924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:19.162857 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.739366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:19.739315 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.953172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:19.953137 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.133592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:20.133513 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.276266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:20.276162 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.401568 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:20.401493 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.883057 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:31:20.883002 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:31:21.443085 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:21.443018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.248093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:22.248041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.535161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:22.535074 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw"] Oct 30 19:31:22.608367 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:22.608332 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.734332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:22.734297 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.351185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:23.351121 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.929082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:23.929046 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl"] Oct 30 19:31:23.929692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:23.929287 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" podUID="bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" containerName="maas-api" containerID="cri-o://f56332301430a6599bc7a7768108fc2d475dbea54625997980b00abf6bdace93" gracePeriod=30 Oct 30 19:31:23.979196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:23.979161 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.452006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.422500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.556737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.556675 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.574783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.574754 6514 generic.go:356] "Generic (PLEG): container finished" podID="bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" containerID="f56332301430a6599bc7a7768108fc2d475dbea54625997980b00abf6bdace93" exitCode=0 Oct 30 19:31:24.574962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.574787 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" event={"ID":"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9","Type":"ContainerDied","Data":"f56332301430a6599bc7a7768108fc2d475dbea54625997980b00abf6bdace93"} Oct 30 19:31:24.605517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.605494 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:31:24.667438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.667408 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.706804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.706745 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\") pod \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " Oct 30 19:31:24.706951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.706798 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs\") pod \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " Oct 30 19:31:24.707089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.707063 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca\") pod \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " Oct 30 19:31:24.707220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.707158 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh\" (UniqueName: \"kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh\") pod \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\" (UID: \"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9\") " Oct 30 19:31:24.707410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.707381 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca" pod "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" (UID: "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:24.708084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.708060 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.708997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.708975 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls" pod "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" (UID: "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:24.709104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.709078 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs" pod "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" (UID: "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:24.709202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.709185 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh" pod "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" (UID: "bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9"). InnerVolumeSpecName "kube-api-access-62cmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:24.812888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.812095 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62cmh\" (UniqueName: \"kubernetes.io/projected/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-kube-api-access-62cmh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.812888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.812137 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.812888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:24.812151 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:25.155983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.155566 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b"] Oct 30 19:31:25.155983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.155821 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" podUID="766172a6-4e89-4544-b021-6ea59c0c7789" containerName="istio-proxy" containerID="cri-o://2aa349aa90dc52294905a76baa76378ab4ab70a58d980bd88a52431a23a83bb3" gracePeriod=30 Oct 30 19:31:25.220071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.220040 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.227180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.227157 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz"] Oct 30 19:31:25.260964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.260919 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk"] Oct 30 19:31:25.336359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.336331 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.461553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.461520 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.495676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.495647 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7"] Oct 30 19:31:25.496417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.496394 6514 cpu_manager.go:414] "containerMap: removing container" podUID="bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" containerName="maas-api" Oct 30 19:31:25.496417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.496410 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" containerName="maas-api" Oct 30 19:31:25.496558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.496547 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" containerName="maas-api" Oct 30 19:31:25.500396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.500378 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.509465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.508977 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-229db90d-serving-cert\"" Oct 30 19:31:25.514358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.514335 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7"] Oct 30 19:31:25.583774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.583748 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" Oct 30 19:31:25.583928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.583740 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl" event={"ID":"bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9","Type":"ContainerDied","Data":"9d6c53f260893f381b6609c53c81b5128a9b30145dc1a658f97ab6a945edc8c9"} Oct 30 19:31:25.583928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.583903 6514 scope.go:122] "RemoveContainer" containerID="f56332301430a6599bc7a7768108fc2d475dbea54625997980b00abf6bdace93" Oct 30 19:31:25.599111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.599089 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.609913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.609893 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl"] Oct 30 19:31:25.613915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.613896 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ae6628f-7cb47c8c4f-hvvwl"] Oct 30 19:31:25.627465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.627442 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.627547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.627493 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xfk\" (UniqueName: \"kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.627636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.627613 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.627814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.627795 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.732935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.732122 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.732935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.732195 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xfk\" (UniqueName: \"kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.732935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.732278 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.732935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.732468 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.739896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.733421 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.752887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.750112 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.838886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.827301 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.854888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.839440 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:25.886695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:25.878468 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xfk\" (UniqueName: \"kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk\") pod \"maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:26.030715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.030636 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.122633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.122602 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:26.162681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.162652 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9" path="/var/lib/kubelet/pods/bbf6f95f-0e6c-445d-b40e-b2e6d0ffc7f9/volumes" Oct 30 19:31:26.242025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.241974 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.380966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.375703 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.458009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.457908 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7"] Oct 30 19:31:26.516472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.516440 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.627890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.627118 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" event={"ID":"403fd967-5dc4-4f47-9d6b-0907ff757439","Type":"ContainerStarted","Data":"02d8f48a73905929f662192d4efb3689d7eac3aba12ca23a66b5c7017b30da79"} Oct 30 19:31:26.937828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:26.937802 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.199811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.199782 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.495395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.495312 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j"] Oct 30 19:31:27.500510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.500475 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.528723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.528686 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j"] Oct 30 19:31:27.536225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.536188 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" podUID="1380c68e-9976-4720-ac62-a167f0e5dc60" containerName="payload-pre-processing" containerID="cri-o://9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71" gracePeriod=25 Oct 30 19:31:27.550537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.550506 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb"] Oct 30 19:31:27.560874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.560725 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb"] Oct 30 19:31:27.560962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.560872 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.606661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.606619 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.606922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.606678 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.606922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.606701 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609043 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609102 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609136 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609156 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btv6\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609193 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64blt\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609266 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609350 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609385 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609406 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609525 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609557 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609582 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609603 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609617 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.611162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.609676 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.635794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.635764 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" event={"ID":"403fd967-5dc4-4f47-9d6b-0907ff757439","Type":"ContainerStarted","Data":"c58a5728daabfeb99ec6080370068d4c81a1d2ee4eb3e7780030dbda4a46e3d0"} Oct 30 19:31:27.635964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.635942 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:27.659301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.659260 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" podStartSLOduration=2.268151033 podStartE2EDuration="2.65924745s" totalImagesPullingTime="391.096417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:27.658147662 +0000 UTC m=+857.615966849" watchObservedRunningTime="2026-10-30 19:31:27.65924745 +0000 UTC m=+857.617066639" Oct 30 19:31:27.711468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711439 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.711649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711490 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711509 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btv6\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.711649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711534 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64blt\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711551 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711610 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711652 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711677 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711805 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711844 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711904 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711931 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.711983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711974 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.712326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.711999 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712163 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712173 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.712326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712202 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712218 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.712326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712312 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712342 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712390 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712495 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.712763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712621 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.712763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712664 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.712763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712663 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.713016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.712984 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.713096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.713067 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.713435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.713403 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.714977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.714947 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.715481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.715164 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.715481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.715221 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.716168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.716144 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.730393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.730286 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.730393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.730297 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.730562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.730419 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64blt\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.731803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.731773 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btv6\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.830575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.830544 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:27.831747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.831725 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:31:27.834630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.834602 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.922450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.922245 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume\") pod \"1380c68e-9976-4720-ac62-a167f0e5dc60\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " Oct 30 19:31:27.928237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.925126 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd\" (UniqueName: \"kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd\") pod \"1380c68e-9976-4720-ac62-a167f0e5dc60\" (UID: \"1380c68e-9976-4720-ac62-a167f0e5dc60\") " Oct 30 19:31:27.936439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.936194 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:27.960327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.948466 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume" pod "1380c68e-9976-4720-ac62-a167f0e5dc60" (UID: "1380c68e-9976-4720-ac62-a167f0e5dc60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:27.992901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:27.982073 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd" pod "1380c68e-9976-4720-ac62-a167f0e5dc60" (UID: "1380c68e-9976-4720-ac62-a167f0e5dc60"). InnerVolumeSpecName "kube-api-access-t2fkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:28.041354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.041288 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.044724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.044558 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fkd\" (UniqueName: \"kubernetes.io/projected/1380c68e-9976-4720-ac62-a167f0e5dc60-kube-api-access-t2fkd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:28.044724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.044595 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1380c68e-9976-4720-ac62-a167f0e5dc60-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:28.162455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.162421 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.293450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.293419 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j"] Oct 30 19:31:28.293818 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:31:28.293603 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode106113d_7509_4af9_8a5f_adaf0039fc06.slice/crio-fd82fee7b3f300b9e62e553f1a95a30a74c4066c70470c3e9395bc52def6bfd6 WatchSource:0}: Error finding container fd82fee7b3f300b9e62e553f1a95a30a74c4066c70470c3e9395bc52def6bfd6: Status 404 returned error can't find the container with id fd82fee7b3f300b9e62e553f1a95a30a74c4066c70470c3e9395bc52def6bfd6 Oct 30 19:31:28.296346 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.296320 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:28.296406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.296378 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:28.296449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.296410 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:28.307619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.307597 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb"] Oct 30 19:31:28.308164 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:31:28.308142 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6b533e_4850_4148_b43e_a3504dbf8d85.slice/crio-100fbd401ccd2f28b6772372417e836de735fc26eac1ad8813878885f08e6d47 WatchSource:0}: Error finding container 100fbd401ccd2f28b6772372417e836de735fc26eac1ad8813878885f08e6d47: Status 404 returned error can't find the container with id 100fbd401ccd2f28b6772372417e836de735fc26eac1ad8813878885f08e6d47 Oct 30 19:31:28.310524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.310496 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:28.310600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.310562 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:28.310600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.310590 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:28.353253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.353089 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:31:28.518672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.518636 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.647804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.647732 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.648006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.647950 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" Oct 30 19:31:28.648077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.647846 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" event={"ID":"1380c68e-9976-4720-ac62-a167f0e5dc60","Type":"ContainerDied","Data":"9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71"} Oct 30 19:31:28.648130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.647819 6514 generic.go:356] "Generic (PLEG): container finished" podID="1380c68e-9976-4720-ac62-a167f0e5dc60" containerID="9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71" exitCode=0 Oct 30 19:31:28.648225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.648211 6514 scope.go:122] "RemoveContainer" containerID="9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71" Oct 30 19:31:28.648400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.648377 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw" event={"ID":"1380c68e-9976-4720-ac62-a167f0e5dc60","Type":"ContainerDied","Data":"3eb5a6f986ae218ca305e70539912c36dfb247c5b29c5c86d06708242d4527c2"} Oct 30 19:31:28.651084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.651021 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" event={"ID":"8b6b533e-4850-4148-b43e-a3504dbf8d85","Type":"ContainerStarted","Data":"f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5"} Oct 30 19:31:28.651208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.651087 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" event={"ID":"8b6b533e-4850-4148-b43e-a3504dbf8d85","Type":"ContainerStarted","Data":"100fbd401ccd2f28b6772372417e836de735fc26eac1ad8813878885f08e6d47"} Oct 30 19:31:28.654836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.654807 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" event={"ID":"e106113d-7509-4af9-8a5f-adaf0039fc06","Type":"ContainerStarted","Data":"c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571"} Oct 30 19:31:28.655068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.654853 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" event={"ID":"e106113d-7509-4af9-8a5f-adaf0039fc06","Type":"ContainerStarted","Data":"fd82fee7b3f300b9e62e553f1a95a30a74c4066c70470c3e9395bc52def6bfd6"} Oct 30 19:31:28.720885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.714472 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podStartSLOduration=1.714453225 podStartE2EDuration="1.714453225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:28.674642688 +0000 UTC m=+858.632461877" watchObservedRunningTime="2026-10-30 19:31:28.714453225 +0000 UTC m=+858.672272414" Oct 30 19:31:28.735911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.732142 6514 scope.go:122] "RemoveContainer" containerID="9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71" Oct 30 19:31:28.749816 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:31:28.749760 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71\": container with ID starting with 9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71 not found: ID does not exist" containerID="9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71" Oct 30 19:31:28.751492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.751454 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71"} err="failed to get container status \"9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71\": rpc error: code = NotFound desc = could not find container \"9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71\": container with ID starting with 9d1eeaa2630784098a43aff1343020bdf0c5fb1b3114e35ae33d8cd64f60fb71 not found: ID does not exist" Oct 30 19:31:28.861036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.861000 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:28.868953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.868930 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw"] Oct 30 19:31:28.876048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.876003 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ae6628f-6fbdd54f79-qh5cw"] Oct 30 19:31:28.876621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.876566 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podStartSLOduration=1.876550522 podStartE2EDuration="1.876550522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:28.785639821 +0000 UTC m=+858.743459010" watchObservedRunningTime="2026-10-30 19:31:28.876550522 +0000 UTC m=+858.834369710" Oct 30 19:31:28.971887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:28.964385 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:29.071887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.061172 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.379941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.375948 6514 patch_prober.go:29] interesting pod/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:31:29.379941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.376014 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" podUID="766172a6-4e89-4544-b021-6ea59c0c7789" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:31:29.492888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.488642 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.844889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.843302 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:29.844889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.843373 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:29.952360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.952315 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:29.952523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:29.952388 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:30.164711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.164683 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1380c68e-9976-4720-ac62-a167f0e5dc60" path="/var/lib/kubelet/pods/1380c68e-9976-4720-ac62-a167f0e5dc60/volumes" Oct 30 19:31:30.241970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.241017 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" podUID="e8156308-0fbb-4b06-8b85-2a6bee8729f4" containerName="payload-processing" containerID="cri-o://eedea8b85d9ac5afba9f694da6cf568c98ce65573995b5a82973a157229d8bc2" gracePeriod=25 Oct 30 19:31:30.247744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.247709 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:30.261796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.261758 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" podUID="bdde9ebb-2a7a-4cca-95f3-d390b8947867" containerName="payload-pre-processing" containerID="cri-o://4b0c64c6d6581327290e35df8dc0545f02cc150090c35073a87fd770945f701a" gracePeriod=25 Oct 30 19:31:30.693144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.692940 6514 generic.go:356] "Generic (PLEG): container finished" podID="e8156308-0fbb-4b06-8b85-2a6bee8729f4" containerID="eedea8b85d9ac5afba9f694da6cf568c98ce65573995b5a82973a157229d8bc2" exitCode=0 Oct 30 19:31:30.693144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.693065 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" event={"ID":"e8156308-0fbb-4b06-8b85-2a6bee8729f4","Type":"ContainerDied","Data":"eedea8b85d9ac5afba9f694da6cf568c98ce65573995b5a82973a157229d8bc2"} Oct 30 19:31:30.695904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.695644 6514 generic.go:356] "Generic (PLEG): container finished" podID="bdde9ebb-2a7a-4cca-95f3-d390b8947867" containerID="4b0c64c6d6581327290e35df8dc0545f02cc150090c35073a87fd770945f701a" exitCode=0 Oct 30 19:31:30.695904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.695815 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" event={"ID":"bdde9ebb-2a7a-4cca-95f3-d390b8947867","Type":"ContainerDied","Data":"4b0c64c6d6581327290e35df8dc0545f02cc150090c35073a87fd770945f701a"} Oct 30 19:31:30.698166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.698142 6514 generic.go:356] "Generic (PLEG): container finished" podID="766172a6-4e89-4544-b021-6ea59c0c7789" containerID="2aa349aa90dc52294905a76baa76378ab4ab70a58d980bd88a52431a23a83bb3" exitCode=0 Oct 30 19:31:30.699282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.699257 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" event={"ID":"766172a6-4e89-4544-b021-6ea59c0c7789","Type":"ContainerDied","Data":"2aa349aa90dc52294905a76baa76378ab4ab70a58d980bd88a52431a23a83bb3"} Oct 30 19:31:30.850889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.850396 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:30.850889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.850458 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:31.001003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:30.965895 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:31:31.051242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.051067 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.051476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.051457 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.051638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.051623 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.051829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.051813 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.052017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.052002 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.052161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.052146 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.052308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.052295 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.052434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.052418 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.052615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.052595 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert\") pod \"766172a6-4e89-4544-b021-6ea59c0c7789\" (UID: \"766172a6-4e89-4544-b021-6ea59c0c7789\") " Oct 30 19:31:31.053938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.053916 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:31.054581 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.054559 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:31.077764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.077734 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:31.085064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.083783 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:31.085064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.083876 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:31.086348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.085950 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:31.086348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.086240 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:31.086348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.086296 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:31.087107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.087050 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:31.088788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.088739 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:31:31.090796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.090764 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx" pod "766172a6-4e89-4544-b021-6ea59c0c7789" (UID: "766172a6-4e89-4544-b021-6ea59c0c7789"). InnerVolumeSpecName "kube-api-access-4qxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159605 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159639 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766172a6-4e89-4544-b021-6ea59c0c7789-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159655 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159668 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159683 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159697 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159711 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766172a6-4e89-4544-b021-6ea59c0c7789-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159725 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxmx\" (UniqueName: \"kubernetes.io/projected/766172a6-4e89-4544-b021-6ea59c0c7789-kube-api-access-4qxmx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.160918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.159741 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766172a6-4e89-4544-b021-6ea59c0c7789-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.254182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.254068 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:31.272371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.272346 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:31:31.276042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.276017 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:31:31.365280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.365251 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume\") pod \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " Oct 30 19:31:31.365443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.365316 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume\") pod \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " Oct 30 19:31:31.365443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.365363 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5\" (UniqueName: \"kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5\") pod \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\" (UID: \"e8156308-0fbb-4b06-8b85-2a6bee8729f4\") " Oct 30 19:31:31.365553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.365493 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc\" (UniqueName: \"kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc\") pod \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\" (UID: \"bdde9ebb-2a7a-4cca-95f3-d390b8947867\") " Oct 30 19:31:31.366596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.366568 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume" pod "e8156308-0fbb-4b06-8b85-2a6bee8729f4" (UID: "e8156308-0fbb-4b06-8b85-2a6bee8729f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:31.366953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.366932 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume" pod "bdde9ebb-2a7a-4cca-95f3-d390b8947867" (UID: "bdde9ebb-2a7a-4cca-95f3-d390b8947867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:31.368074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.368049 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc" pod "bdde9ebb-2a7a-4cca-95f3-d390b8947867" (UID: "bdde9ebb-2a7a-4cca-95f3-d390b8947867"). InnerVolumeSpecName "kube-api-access-xmwlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:31.369929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.369907 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5" pod "e8156308-0fbb-4b06-8b85-2a6bee8729f4" (UID: "e8156308-0fbb-4b06-8b85-2a6bee8729f4"). InnerVolumeSpecName "kube-api-access-7npw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:31.482888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.482149 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9ebb-2a7a-4cca-95f3-d390b8947867-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.482888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.482185 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8156308-0fbb-4b06-8b85-2a6bee8729f4-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.482888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.482200 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npw5\" (UniqueName: \"kubernetes.io/projected/e8156308-0fbb-4b06-8b85-2a6bee8729f4-kube-api-access-7npw5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.482888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.482218 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwlc\" (UniqueName: \"kubernetes.io/projected/bdde9ebb-2a7a-4cca-95f3-d390b8947867-kube-api-access-xmwlc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.513746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.513633 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:31.643908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.643853 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:31.844384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.844295 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:31.844963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.844925 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:31.873884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.872215 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:31.946889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.933272 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" event={"ID":"766172a6-4e89-4544-b021-6ea59c0c7789","Type":"ContainerDied","Data":"b3f0582fbea461514c3d3a58962a196904f5b04489b7cb9e77d1595e904c94af"} Oct 30 19:31:31.947455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.943264 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b" Oct 30 19:31:31.976982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.953480 6514 scope.go:122] "RemoveContainer" containerID="2aa349aa90dc52294905a76baa76378ab4ab70a58d980bd88a52431a23a83bb3" Oct 30 19:31:32.011958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.978350 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.011958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:31.978406 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:32.135938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.118727 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" event={"ID":"e8156308-0fbb-4b06-8b85-2a6bee8729f4","Type":"ContainerDied","Data":"43fe0ab562da6cda995b4f995174a4ebc82f3695ea51b12cc29869c35e33646a"} Oct 30 19:31:32.271096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.126118 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz" Oct 30 19:31:32.271830 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.174954 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" Oct 30 19:31:32.446438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.446415 6514 scope.go:122] "RemoveContainer" containerID="eedea8b85d9ac5afba9f694da6cf568c98ce65573995b5a82973a157229d8bc2" Oct 30 19:31:32.459131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.459096 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:32.531090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.519150 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk" event={"ID":"bdde9ebb-2a7a-4cca-95f3-d390b8947867","Type":"ContainerDied","Data":"33e6d974c4520f0696356c22f44ba8d31b6d725f347e74c5ec8ac51ffb632dfa"} Oct 30 19:31:32.535366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.531784 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b"] Oct 30 19:31:32.682896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.679445 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:32.690534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.690512 6514 scope.go:122] "RemoveContainer" containerID="4b0c64c6d6581327290e35df8dc0545f02cc150090c35073a87fd770945f701a" Oct 30 19:31:32.703990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.698394 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2ae6628f-openshift-default-856f6c7c6f-q5n4b"] Oct 30 19:31:32.717039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.717009 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz"] Oct 30 19:31:32.768118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.768090 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77930c58-f86d9d8fd-gg9bz"] Oct 30 19:31:32.782159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.782131 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk"] Oct 30 19:31:32.848911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.841585 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.857889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.849962 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:32.864394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.864342 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77930c58-75d6fb56d7-s8tgk"] Oct 30 19:31:32.872176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.872140 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx"] Oct 30 19:31:32.873881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.873843 6514 cpu_manager.go:414] "containerMap: removing container" podUID="bdde9ebb-2a7a-4cca-95f3-d390b8947867" containerName="payload-pre-processing" Oct 30 19:31:32.873998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.873887 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdde9ebb-2a7a-4cca-95f3-d390b8947867" containerName="payload-pre-processing" Oct 30 19:31:32.873998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.873918 6514 cpu_manager.go:414] "containerMap: removing container" podUID="766172a6-4e89-4544-b021-6ea59c0c7789" containerName="istio-proxy" Oct 30 19:31:32.873998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.873924 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766172a6-4e89-4544-b021-6ea59c0c7789" containerName="istio-proxy" Oct 30 19:31:32.873998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.873952 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e8156308-0fbb-4b06-8b85-2a6bee8729f4" containerName="payload-processing" Oct 30 19:31:32.873998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.873958 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8156308-0fbb-4b06-8b85-2a6bee8729f4" containerName="payload-processing" Oct 30 19:31:32.874240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.874008 6514 cpu_manager.go:414] "containerMap: removing container" podUID="1380c68e-9976-4720-ac62-a167f0e5dc60" containerName="payload-pre-processing" Oct 30 19:31:32.874240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.874013 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1380c68e-9976-4720-ac62-a167f0e5dc60" containerName="payload-pre-processing" Oct 30 19:31:32.874240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.874212 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8156308-0fbb-4b06-8b85-2a6bee8729f4" containerName="payload-processing" Oct 30 19:31:32.874240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.874229 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="1380c68e-9976-4720-ac62-a167f0e5dc60" containerName="payload-pre-processing" Oct 30 19:31:32.874418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.874244 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdde9ebb-2a7a-4cca-95f3-d390b8947867" containerName="payload-pre-processing" Oct 30 19:31:32.874418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.874252 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="766172a6-4e89-4544-b021-6ea59c0c7789" containerName="istio-proxy" Oct 30 19:31:32.882351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.882320 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx"] Oct 30 19:31:32.882481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.882464 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:32.886883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.886070 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:32.887135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.887114 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5cb1c5cd-openshift-default-dockercfg-r782f\"" Oct 30 19:31:32.968454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.949720 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.968720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:32.968682 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:33.196887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.099116 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.198173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.198133 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.198433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.198415 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.198588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.198571 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.241948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.226787 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.296884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.285131 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.296884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.285183 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jz46\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.296884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.285215 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.296884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.285357 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.352926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.336286 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387237 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387401 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387472 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387530 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387641 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387681 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387725 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387941 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.389911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.387973 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jz46\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.421128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.416231 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.425457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.421571 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.425457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.425013 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.426248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.426183 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.426611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.426587 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.450189 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.450157 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.476998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.475927 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jz46\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.477787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.477735 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.478381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.478336 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token\") pod \"e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.575558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.575477 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:33.586020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.577372 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:33.669694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.669663 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:33.740096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.739841 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:33.883092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.882980 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.885658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.884996 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:33.958923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.950308 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.958923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:33.950375 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:34.162470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.162399 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766172a6-4e89-4544-b021-6ea59c0c7789" path="/var/lib/kubelet/pods/766172a6-4e89-4544-b021-6ea59c0c7789/volumes" Oct 30 19:31:34.162853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.162840 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdde9ebb-2a7a-4cca-95f3-d390b8947867" path="/var/lib/kubelet/pods/bdde9ebb-2a7a-4cca-95f3-d390b8947867/volumes" Oct 30 19:31:34.163319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.163304 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8156308-0fbb-4b06-8b85-2a6bee8729f4" path="/var/lib/kubelet/pods/e8156308-0fbb-4b06-8b85-2a6bee8729f4/volumes" Oct 30 19:31:34.252952 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:31:34.240119 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c5f79c_321e_49a1_9382_9f1dde2fbc37.slice/crio-adbb8766fa5cf94f776a3801b4e121fc3c6dc92233cec27534b6e9d22f6ebb95 WatchSource:0}: Error finding container adbb8766fa5cf94f776a3801b4e121fc3c6dc92233cec27534b6e9d22f6ebb95: Status 404 returned error can't find the container with id adbb8766fa5cf94f776a3801b4e121fc3c6dc92233cec27534b6e9d22f6ebb95 Oct 30 19:31:34.252952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.246020 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:34.252952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.246144 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:34.252952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.246214 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:34.252952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.251517 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx"] Oct 30 19:31:34.285569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.285536 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" event={"ID":"34c5f79c-321e-49a1-9382-9f1dde2fbc37","Type":"ContainerStarted","Data":"adbb8766fa5cf94f776a3801b4e121fc3c6dc92233cec27534b6e9d22f6ebb95"} Oct 30 19:31:34.481272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.481243 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:34.873552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.870974 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:34.873552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.871055 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:34.959952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.958654 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:34.959952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:34.958710 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:35.170154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.169783 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:35.262725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.261682 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz"] Oct 30 19:31:35.286401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.285939 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz"] Oct 30 19:31:35.286401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.286078 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.341904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.317359 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-395ccc-dockercfg-v4s55\"" Oct 30 19:31:35.341904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.317505 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-395ccc\"" Oct 30 19:31:35.341904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.324759 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:35.484564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.484296 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.484564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.484407 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sd2k\" (UniqueName: \"kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k\") pod \"payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.659888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.639138 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.660288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.660240 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sd2k\" (UniqueName: \"kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k\") pod \"payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.677995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.643781 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.789908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.789774 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" event={"ID":"34c5f79c-321e-49a1-9382-9f1dde2fbc37","Type":"ContainerStarted","Data":"40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6"} Oct 30 19:31:35.847899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.836747 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:35.847899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.836799 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:35.847899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.840378 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sd2k\" (UniqueName: \"kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k\") pod \"payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:35.847899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.840577 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" podStartSLOduration=3.840562637 podStartE2EDuration="3.840562637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:35.836470397 +0000 UTC m=+865.794289584" watchObservedRunningTime="2026-10-30 19:31:35.840562637 +0000 UTC m=+865.798381826" Oct 30 19:31:35.970886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.961148 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:35.970886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:35.961203 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:36.053993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.052142 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:36.438709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.438464 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:36.600936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.591219 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:36.617049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.606211 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz"] Oct 30 19:31:36.620211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.620185 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:36.798900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.798842 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" event={"ID":"dc005260-0822-4c07-a4b0-caae7db8468c","Type":"ContainerStarted","Data":"45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e"} Oct 30 19:31:36.799090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.798923 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" event={"ID":"dc005260-0822-4c07-a4b0-caae7db8468c","Type":"ContainerStarted","Data":"633eb9e17a5cfb32857fbc7b44946ed52f8858007e3b1018b75cca044c93b75b"} Oct 30 19:31:36.799215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.799194 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:36.799268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.799244 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:36.800345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.800327 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:31:36.818946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.818898 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" podStartSLOduration=1.818883541 podStartE2EDuration="1.818883541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:36.816578114 +0000 UTC m=+866.774397300" watchObservedRunningTime="2026-10-30 19:31:36.818883541 +0000 UTC m=+866.776702729" Oct 30 19:31:36.831520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.831493 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:31:36.831648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.831548 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 19:31:36.948775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.948738 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:36.948985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:36.948800 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:37.472234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:37.472199 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:37.580397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:37.580364 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:37.699885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:37.699432 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.244788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.244756 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:38.245263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.245243 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:38.246124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.246102 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:31:38.282570 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.282537 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:38.283233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.283211 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:38.284480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.284197 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:31:38.696753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.689141 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.808189 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.808138 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.854816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.854790 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5"] Oct 30 19:31:38.859285 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.859238 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:38.865783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.865162 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5cb1c5cd\"" Oct 30 19:31:38.869108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.869084 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5cb1c5cd-dockercfg-59xb6\"" Oct 30 19:31:38.869605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.869588 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5"] Oct 30 19:31:38.911541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.911514 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb"] Oct 30 19:31:38.916988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.916968 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:38.945706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.945677 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.958266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:38.958221 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb"] Oct 30 19:31:39.019897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.016304 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume\") pod \"payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.019897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.016390 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2sc\" (UniqueName: \"kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc\") pod \"payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.177478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.152114 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume\") pod \"payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.177478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.152189 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7crd\" (UniqueName: \"kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd\") pod \"payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.177478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.152255 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2sc\" (UniqueName: \"kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc\") pod \"payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.177478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.152427 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.177478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.153541 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume\") pod \"payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.251933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.249551 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2sc\" (UniqueName: \"kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc\") pod \"payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.290896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.287936 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.301448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.295946 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7crd\" (UniqueName: \"kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd\") pod \"payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.301448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.297312 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.362884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.359348 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7crd\" (UniqueName: \"kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd\") pod \"payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.468564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.468534 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:39.483960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.483937 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:39.766930 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:31:39.766902 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8e509a_b187_4752_8104_98ed7b34835c.slice/crio-dc56f736944aa196604e7a8b24c5fd6dc27a5781298d7b7afcc1956dd84c1cea WatchSource:0}: Error finding container dc56f736944aa196604e7a8b24c5fd6dc27a5781298d7b7afcc1956dd84c1cea: Status 404 returned error can't find the container with id dc56f736944aa196604e7a8b24c5fd6dc27a5781298d7b7afcc1956dd84c1cea Oct 30 19:31:39.767293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.767178 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb"] Oct 30 19:31:39.785706 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:31:39.785655 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode693a875_05f4_44b3_a715_90736873c473.slice/crio-a1cdb789b448975da34a0547437c9362b38161319e0ca21e1e0d6ae86dbfb4d3 WatchSource:0}: Error finding container a1cdb789b448975da34a0547437c9362b38161319e0ca21e1e0d6ae86dbfb4d3: Status 404 returned error can't find the container with id a1cdb789b448975da34a0547437c9362b38161319e0ca21e1e0d6ae86dbfb4d3 Oct 30 19:31:39.787299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.787271 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5"] Oct 30 19:31:39.830968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.830945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:39.978897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.976129 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" event={"ID":"e693a875-05f4-44b3-a715-90736873c473","Type":"ContainerStarted","Data":"a1cdb789b448975da34a0547437c9362b38161319e0ca21e1e0d6ae86dbfb4d3"} Oct 30 19:31:39.984926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.979523 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" event={"ID":"8f8e509a-b187-4752-8104-98ed7b34835c","Type":"ContainerStarted","Data":"c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779"} Oct 30 19:31:39.984926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.979579 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" event={"ID":"8f8e509a-b187-4752-8104-98ed7b34835c","Type":"ContainerStarted","Data":"dc56f736944aa196604e7a8b24c5fd6dc27a5781298d7b7afcc1956dd84c1cea"} Oct 30 19:31:39.984926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:39.980558 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:40.014119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:40.014061 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" podStartSLOduration=2.013272556 podStartE2EDuration="2.013272556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:40.006806114 +0000 UTC m=+869.964625304" watchObservedRunningTime="2026-10-30 19:31:40.013272556 +0000 UTC m=+869.971091745" Oct 30 19:31:40.756800 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:40.756709 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.989521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:40.989478 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" event={"ID":"e693a875-05f4-44b3-a715-90736873c473","Type":"ContainerStarted","Data":"1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f"} Oct 30 19:31:40.989951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:40.989568 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:41.013310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:41.013234 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" podStartSLOduration=3.013222375 podStartE2EDuration="3.013222375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:41.007363107 +0000 UTC m=+870.965182298" watchObservedRunningTime="2026-10-30 19:31:41.013222375 +0000 UTC m=+870.971041563" Oct 30 19:31:41.716248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:41.716220 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.530799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:42.530765 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.336779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:43.336753 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.974542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:43.974510 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp"] Oct 30 19:31:43.980505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:43.980483 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:43.993694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:43.991091 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp"] Oct 30 19:31:43.993694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:43.992161 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-395ccc-serving-cert\"" Oct 30 19:31:44.042642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.042611 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:44.052792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.052764 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.053061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.053044 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvv2g\" (UniqueName: \"kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.053110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.053084 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.053252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.053231 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.167200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.162472 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvv2g\" (UniqueName: \"kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.167200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.162561 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.167200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.162679 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.167200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.162927 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.167200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.164320 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.201909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.195471 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:44.222918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.213624 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.246929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.241642 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.250133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.250061 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvv2g\" (UniqueName: \"kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g\") pod \"maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.312884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.312649 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:44.586945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.532672 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:31:44.806145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:44.801032 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp"] Oct 30 19:31:45.096839 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:45.096732 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" event={"ID":"52e5bfbd-00da-4e97-96d5-ab8a2683664e","Type":"ContainerStarted","Data":"bfce4015f57f4815c973e2fedec766b97ea2a103d9ae7a1b9fcf839a4795dc0d"} Oct 30 19:31:45.524485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:45.524455 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:46.179803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:46.179515 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" event={"ID":"52e5bfbd-00da-4e97-96d5-ab8a2683664e","Type":"ContainerStarted","Data":"7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a"} Oct 30 19:31:46.180530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:46.180509 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:46.235891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:46.232674 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" podStartSLOduration=2.666835094 podStartE2EDuration="3.232654635s" totalImagesPullingTime="565.819541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:46.197275362 +0000 UTC m=+876.155094552" watchObservedRunningTime="2026-10-30 19:31:46.232654635 +0000 UTC m=+876.190473823" Oct 30 19:31:47.701893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:47.698962 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:47.746888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:47.743891 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:47.910891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:47.910414 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:31:48.001942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:47.960327 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.051886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.049431 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.214897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.214347 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.561664 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.561625 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:31:48.711324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.711237 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:48.730998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.712591 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:31:48.885339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.817146 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.885339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.874970 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:31:48.885339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.877317 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:48.968288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.967151 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.022973 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:48.968758 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:31:49.126887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.126765 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.164940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.130465 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:31:49.164940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.146006 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.360913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.290815 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:49.360913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.291146 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:31:49.396300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.384079 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:49.717657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.717600 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:31:49.723570 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.723522 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:31:49.733199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.733154 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.848889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:49.834134 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.107979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.030468 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:50.182893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.156233 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:50.354556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.354512 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:50.432223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.432180 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.738953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.735985 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.852994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.848051 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:31:50.969890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.867783 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.046937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:50.998421 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:31:51.309688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.037301 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.317836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.053013 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.318183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.115997 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:51.318335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.318309 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.332233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.293168 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.357109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.357031 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.452504 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:31:51.452463 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:31:51.487888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.468756 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:31:51.501921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.491008 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:31:51.501921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.472937 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.630001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.628114 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:31:51.676071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.675831 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:31:51.676071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.675913 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:31:51.676071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.675949 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" Oct 30 19:31:51.686270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:51.686240 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:31:52.080224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.080191 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:31:52.084348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.084225 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:31:52.115477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.115444 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:52.251952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.251922 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:31:52.974351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.974318 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4"] Oct 30 19:31:52.980513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.980488 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.009886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:52.992701 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4"] Oct 30 19:31:53.009886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.002737 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5cb1c5cd-serving-cert\"" Oct 30 19:31:53.046275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.046245 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:53.147939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.144086 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.147939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.144321 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79m9\" (UniqueName: \"kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.147939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.144362 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.147939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.144580 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.171149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.171116 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.250130 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79m9\" (UniqueName: \"kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.250186 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.250395 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.250472 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:31:53.250713 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-serving-cert: secret "maas-api-e2e-derive-5cb1c5cd-serving-cert" not found Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:31:53.250788 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls podName:c58c0949-fb4e-4b2e-ae1e-4f049df6f401 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:53.750766955 +0000 UTC m=+883.708586137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls") pod "maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" (UID: "c58c0949-fb4e-4b2e-ae1e-4f049df6f401") : secret "maas-api-e2e-derive-5cb1c5cd-serving-cert" not found Oct 30 19:31:53.252932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.251993 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.264260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.264222 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.272257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.272144 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79m9\" (UniqueName: \"kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.433615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.433564 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:53.572956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.560804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:53.782887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.782137 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.812040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.811989 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\") pod \"maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:53.932995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:53.932684 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:54.064549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:54.064469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:54.303396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:54.293410 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4"] Oct 30 19:31:54.354287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:54.354249 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" event={"ID":"c58c0949-fb4e-4b2e-ae1e-4f049df6f401","Type":"ContainerStarted","Data":"81ac7376c2a923961a7e1752fe4bf14c822f7f2f61e8e7f5023dacfced571f9c"} Oct 30 19:31:54.536309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:54.536269 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb"] Oct 30 19:31:54.625594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:54.625509 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:54.735484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:54.735447 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:55.365656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:55.365616 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" event={"ID":"c58c0949-fb4e-4b2e-ae1e-4f049df6f401","Type":"ContainerStarted","Data":"ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe"} Oct 30 19:31:55.366572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:55.365724 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:31:55.385487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:55.385447 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" podStartSLOduration=2.862874903 podStartE2EDuration="3.385433761s" totalImagesPullingTime="522.558858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:55.383514061 +0000 UTC m=+885.341333251" watchObservedRunningTime="2026-10-30 19:31:55.385433761 +0000 UTC m=+885.343252965" Oct 30 19:31:55.446670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:55.446633 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.100087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.100030 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7"] Oct 30 19:31:56.100350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.100295 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" podUID="403fd967-5dc4-4f47-9d6b-0907ff757439" containerName="maas-api" containerID="cri-o://c58a5728daabfeb99ec6080370068d4c81a1d2ee4eb3e7780030dbda4a46e3d0" gracePeriod=30 Oct 30 19:31:56.146536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.146501 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.306811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.306651 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.630196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.630147 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" event={"ID":"403fd967-5dc4-4f47-9d6b-0907ff757439","Type":"ContainerDied","Data":"c58a5728daabfeb99ec6080370068d4c81a1d2ee4eb3e7780030dbda4a46e3d0"} Oct 30 19:31:56.644764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.565985 6514 generic.go:356] "Generic (PLEG): container finished" podID="403fd967-5dc4-4f47-9d6b-0907ff757439" containerID="c58a5728daabfeb99ec6080370068d4c81a1d2ee4eb3e7780030dbda4a46e3d0" exitCode=0 Oct 30 19:31:56.694920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.694882 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.735181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.734889 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:56.794886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.790360 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs\") pod \"403fd967-5dc4-4f47-9d6b-0907ff757439\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " Oct 30 19:31:56.794886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.790447 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls\") pod \"403fd967-5dc4-4f47-9d6b-0907ff757439\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " Oct 30 19:31:56.794886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.790666 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca\") pod \"403fd967-5dc4-4f47-9d6b-0907ff757439\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " Oct 30 19:31:56.794886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.790816 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk\" (UniqueName: \"kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk\") pod \"403fd967-5dc4-4f47-9d6b-0907ff757439\" (UID: \"403fd967-5dc4-4f47-9d6b-0907ff757439\") " Oct 30 19:31:56.866836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.866647 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls" pod "403fd967-5dc4-4f47-9d6b-0907ff757439" (UID: "403fd967-5dc4-4f47-9d6b-0907ff757439"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:56.866836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.866673 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs" pod "403fd967-5dc4-4f47-9d6b-0907ff757439" (UID: "403fd967-5dc4-4f47-9d6b-0907ff757439"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:56.866836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.866802 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk" pod "403fd967-5dc4-4f47-9d6b-0907ff757439" (UID: "403fd967-5dc4-4f47-9d6b-0907ff757439"). InnerVolumeSpecName "kube-api-access-79xfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:56.867558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.867515 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca" pod "403fd967-5dc4-4f47-9d6b-0907ff757439" (UID: "403fd967-5dc4-4f47-9d6b-0907ff757439"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:56.872495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.872462 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.904642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.902584 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:56.904642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.902616 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403fd967-5dc4-4f47-9d6b-0907ff757439-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:56.904642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.902626 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403fd967-5dc4-4f47-9d6b-0907ff757439-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:56.904642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:56.902634 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79xfk\" (UniqueName: \"kubernetes.io/projected/403fd967-5dc4-4f47-9d6b-0907ff757439-kube-api-access-79xfk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:57.722403 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:57.722285 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:57.766288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:57.752371 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" event={"ID":"403fd967-5dc4-4f47-9d6b-0907ff757439","Type":"ContainerDied","Data":"02d8f48a73905929f662192d4efb3689d7eac3aba12ca23a66b5c7017b30da79"} Oct 30 19:31:57.766288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:57.752449 6514 scope.go:122] "RemoveContainer" containerID="c58a5728daabfeb99ec6080370068d4c81a1d2ee4eb3e7780030dbda4a46e3d0" Oct 30 19:31:57.766288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:57.752585 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7" Oct 30 19:31:57.811939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:57.811908 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7"] Oct 30 19:31:57.824881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:57.821653 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-229db90d-694fb7d5bf-wmgf7"] Oct 30 19:31:58.172764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:58.172658 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403fd967-5dc4-4f47-9d6b-0907ff757439" path="/var/lib/kubelet/pods/403fd967-5dc4-4f47-9d6b-0907ff757439/volumes" Oct 30 19:31:58.190399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:58.190365 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28"] Oct 30 19:31:58.190621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:58.190596 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" containerID="cri-o://dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab" gracePeriod=30 Oct 30 19:31:58.258386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:58.258357 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:58.361472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:58.361426 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.276970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:59.276935 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.402442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:59.396702 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.536616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:59.536537 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" podUID="10aa3ab8-028e-4b43-a283-e2082f5c7a6c" containerName="payload-pre-processing" containerID="cri-o://70ee22904ae5c02a39ded24114ba0130860dd622e237cd4ea1c521582bc2bc2c" gracePeriod=25 Oct 30 19:31:59.790014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:59.789934 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.792390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:59.792357 6514 generic.go:356] "Generic (PLEG): container finished" podID="10aa3ab8-028e-4b43-a283-e2082f5c7a6c" containerID="70ee22904ae5c02a39ded24114ba0130860dd622e237cd4ea1c521582bc2bc2c" exitCode=0 Oct 30 19:31:59.792618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:31:59.792591 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" event={"ID":"10aa3ab8-028e-4b43-a283-e2082f5c7a6c","Type":"ContainerDied","Data":"70ee22904ae5c02a39ded24114ba0130860dd622e237cd4ea1c521582bc2bc2c"} Oct 30 19:32:00.101898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.092535 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:32:00.187888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.187492 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc\" (UniqueName: \"kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc\") pod \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " Oct 30 19:32:00.188082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.187910 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume\") pod \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\" (UID: \"10aa3ab8-028e-4b43-a283-e2082f5c7a6c\") " Oct 30 19:32:00.231887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.229815 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume" pod "10aa3ab8-028e-4b43-a283-e2082f5c7a6c" (UID: "10aa3ab8-028e-4b43-a283-e2082f5c7a6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:00.235407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.235371 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc" pod "10aa3ab8-028e-4b43-a283-e2082f5c7a6c" (UID: "10aa3ab8-028e-4b43-a283-e2082f5c7a6c"). InnerVolumeSpecName "kube-api-access-965sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:00.244435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.244408 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:00.285633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.285552 6514 patch_prober.go:29] interesting pod/e2e-ait-229db90d-openshift-default-8696475765-tkr28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:00.285633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.285610 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:00.290375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.290351 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.290492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.290483 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-965sc\" (UniqueName: \"kubernetes.io/projected/10aa3ab8-028e-4b43-a283-e2082f5c7a6c-kube-api-access-965sc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.395423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.395326 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:00.512518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.512467 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:00.803748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.803712 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" Oct 30 19:32:00.803929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.803712 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb" event={"ID":"10aa3ab8-028e-4b43-a283-e2082f5c7a6c","Type":"ContainerDied","Data":"732fac892b83cb26bdc19a279a99ff3bef944b15657a76b6469ec8e2dfcdd809"} Oct 30 19:32:00.803929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.803910 6514 scope.go:122] "RemoveContainer" containerID="70ee22904ae5c02a39ded24114ba0130860dd622e237cd4ea1c521582bc2bc2c" Oct 30 19:32:00.825821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.825794 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb"] Oct 30 19:32:00.833122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:00.833098 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-229db90d-7b97c78b67-wn2bb"] Oct 30 19:32:01.069626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:01.069531 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.551771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:01.551665 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.661154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:01.661119 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.664949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:01.664914 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:32:01.789536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:01.789493 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.940939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:01.934364 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:02.270275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.270117 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10aa3ab8-028e-4b43-a283-e2082f5c7a6c" path="/var/lib/kubelet/pods/10aa3ab8-028e-4b43-a283-e2082f5c7a6c/volumes" Oct 30 19:32:02.691052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.691011 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:32:02.811466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811428 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.811654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811488 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.811654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811526 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.811654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811641 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.811794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811724 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.811848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811791 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:02.811975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811954 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.812059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.812033 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.812116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.811951 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:02.812116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.812081 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.812211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.812134 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data\") pod \"591aa71d-e36f-42ee-8299-1569c5737b08\" (UID: \"591aa71d-e36f-42ee-8299-1569c5737b08\") " Oct 30 19:32:02.812372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.812351 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:02.812679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.812651 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:02.813042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.813022 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.813153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.813051 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.813153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.813065 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591aa71d-e36f-42ee-8299-1569c5737b08-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.813153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.813076 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.813153 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.813027 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:02.814841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.814809 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:02.814999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.814889 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:02.814999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.814970 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:02.815454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.815429 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg" pod "591aa71d-e36f-42ee-8299-1569c5737b08" (UID: "591aa71d-e36f-42ee-8299-1569c5737b08"). InnerVolumeSpecName "kube-api-access-p4bgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:02.825398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.825347 6514 generic.go:356] "Generic (PLEG): container finished" podID="591aa71d-e36f-42ee-8299-1569c5737b08" containerID="dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab" exitCode=0 Oct 30 19:32:02.825398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.825392 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" Oct 30 19:32:02.825519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.825428 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" event={"ID":"591aa71d-e36f-42ee-8299-1569c5737b08","Type":"ContainerDied","Data":"dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab"} Oct 30 19:32:02.825519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.825482 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28" event={"ID":"591aa71d-e36f-42ee-8299-1569c5737b08","Type":"ContainerDied","Data":"a8f83d7ce822245a00278a60e037d955183027e51058e72e6ab5c7412b10623d"} Oct 30 19:32:02.825519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.825511 6514 scope.go:122] "RemoveContainer" containerID="dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab" Oct 30 19:32:02.847009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.846982 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:02.852988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.850814 6514 scope.go:122] "RemoveContainer" containerID="dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab" Oct 30 19:32:02.852988 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:02.852000 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab\": container with ID starting with dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab not found: ID does not exist" containerID="dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab" Oct 30 19:32:02.852988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.852032 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab"} err="failed to get container status \"dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab\": rpc error: code = NotFound desc = could not find container \"dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab\": container with ID starting with dbcbf8f6e663734ee4422a19678c1cbb29388e530c1afd2c72d873ed086fe5ab not found: ID does not exist" Oct 30 19:32:02.867696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.867674 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28"] Oct 30 19:32:02.874450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.874430 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-229db90d-openshift-default-8696475765-tkr28"] Oct 30 19:32:02.947886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.944951 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bgg\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-kube-api-access-p4bgg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.947886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.944993 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591aa71d-e36f-42ee-8299-1569c5737b08-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.947886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.945011 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.947886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.945025 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591aa71d-e36f-42ee-8299-1569c5737b08-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:02.947886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:02.945040 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591aa71d-e36f-42ee-8299-1569c5737b08-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:03.036829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:03.036795 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:03.161071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:03.160914 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:03.523731 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:03.523576 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:03.951228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:03.951194 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:04.078796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:04.078765 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:04.160600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:04.160567 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" path="/var/lib/kubelet/pods/591aa71d-e36f-42ee-8299-1569c5737b08/volumes" Oct 30 19:32:04.245358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:04.245286 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:04.417054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:04.417018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:04.551972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:04.549527 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:05.621291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:05.621257 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:06.279200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.279169 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:06.325979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.325935 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl"] Oct 30 19:32:06.328333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328311 6514 cpu_manager.go:414] "containerMap: removing container" podUID="403fd967-5dc4-4f47-9d6b-0907ff757439" containerName="maas-api" Oct 30 19:32:06.328333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328333 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403fd967-5dc4-4f47-9d6b-0907ff757439" containerName="maas-api" Oct 30 19:32:06.328471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328348 6514 cpu_manager.go:414] "containerMap: removing container" podUID="10aa3ab8-028e-4b43-a283-e2082f5c7a6c" containerName="payload-pre-processing" Oct 30 19:32:06.328471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328353 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10aa3ab8-028e-4b43-a283-e2082f5c7a6c" containerName="payload-pre-processing" Oct 30 19:32:06.328471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328403 6514 cpu_manager.go:414] "containerMap: removing container" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" Oct 30 19:32:06.328471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328410 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" Oct 30 19:32:06.328596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328502 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="591aa71d-e36f-42ee-8299-1569c5737b08" containerName="istio-proxy" Oct 30 19:32:06.328596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328512 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="403fd967-5dc4-4f47-9d6b-0907ff757439" containerName="maas-api" Oct 30 19:32:06.328596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.328519 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="10aa3ab8-028e-4b43-a283-e2082f5c7a6c" containerName="payload-pre-processing" Oct 30 19:32:06.333979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.333959 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.346226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.346015 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2fd9a937\"" Oct 30 19:32:06.346546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.346520 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2fd9a937-dockercfg-llq8r\"" Oct 30 19:32:06.347910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.347884 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl"] Oct 30 19:32:06.410434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.410396 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume\") pod \"payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.410716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.410693 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szn4l\" (UniqueName: \"kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l\") pod \"payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.441920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.439172 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:06.523888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.523056 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szn4l\" (UniqueName: \"kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l\") pod \"payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.534094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.523339 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume\") pod \"payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.534094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.526084 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume\") pod \"payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.584913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.584709 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szn4l\" (UniqueName: \"kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l\") pod \"payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:06.656891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:06.656354 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:07.074749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:07.074714 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl"] Oct 30 19:32:07.075381 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:07.075353 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5bf917_3608_4595_b00e_5d18c6fc9e97.slice/crio-07a943537b574a2d7c431daefa9432d900654b14a9e8f365e393f917ced9bf9e WatchSource:0}: Error finding container 07a943537b574a2d7c431daefa9432d900654b14a9e8f365e393f917ced9bf9e: Status 404 returned error can't find the container with id 07a943537b574a2d7c431daefa9432d900654b14a9e8f365e393f917ced9bf9e Oct 30 19:32:07.324568 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:07.324537 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:08.069466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:08.069427 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" event={"ID":"8a5bf917-3608-4595-b00e-5d18c6fc9e97","Type":"ContainerStarted","Data":"0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10"} Oct 30 19:32:08.069839 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:08.069498 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" event={"ID":"8a5bf917-3608-4595-b00e-5d18c6fc9e97","Type":"ContainerStarted","Data":"07a943537b574a2d7c431daefa9432d900654b14a9e8f365e393f917ced9bf9e"} Oct 30 19:32:08.070697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:08.070676 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:08.090712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:08.090670 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" podStartSLOduration=2.090658001 podStartE2EDuration="2.090658001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:08.089047231 +0000 UTC m=+898.046866429" watchObservedRunningTime="2026-10-30 19:32:08.090658001 +0000 UTC m=+898.048477189" Oct 30 19:32:08.386546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:08.386469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:09.160593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:09.160559 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:09.304886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:09.304423 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.162044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:10.161999 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.385564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:10.385540 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:32:10.388588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:10.388569 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:32:10.519496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:10.519468 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.733891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:10.709532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:11.054406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:11.054258 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:11.254513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:11.254420 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:12.186721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:12.186683 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:13.044086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:13.044050 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:13.732634 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:13.732604 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:14.448495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:14.448464 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:14.569321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:14.569288 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:14.721766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:14.721689 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.120256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.120157 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf"] Oct 30 19:32:15.154629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.149812 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z"] Oct 30 19:32:15.158882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.155832 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.168886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.164064 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf"] Oct 30 19:32:15.168886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.165132 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.171615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.171468 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z"] Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.243708 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.243757 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.243810 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.243900 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.243943 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbkl\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.243985 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbft\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244020 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244050 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244105 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244143 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244223 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244276 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244306 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244344 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244410 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244439 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244548 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.247887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.244625 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.368178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.363156 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.370467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370362 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.370467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370416 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.370666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370542 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.370666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370652 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.370781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370698 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbkl\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.370781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370745 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbft\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.370905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370783 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.370905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370817 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.370905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370891 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.371091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.370935 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.371091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371055 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.371221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371104 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.371221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371145 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.371221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371194 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.371376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371264 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.371376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371296 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.371486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371417 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.371879 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.372287 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.375618 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.376525 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.376759 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.377469 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.377749 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.377981 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.378182 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.389115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.378411 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.396368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.392563 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.396368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.393564 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.396368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.394936 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.397608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.397004 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.397894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.397843 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbft\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5jm6z\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.398673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.398184 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbkl\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.398673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.398633 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.401094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.401064 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-lnkcf\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.404693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.404668 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:15.540332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.540297 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.673106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.672950 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:15.706744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.706704 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z"] Oct 30 19:32:15.707359 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:15.707335 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf846e02_c35f_4f1c_933f_a5a8a4a64060.slice/crio-72664eeba82d3a180e9c35671cd16b273e0c6a9d8d2b821e5089105a1856430c WatchSource:0}: Error finding container 72664eeba82d3a180e9c35671cd16b273e0c6a9d8d2b821e5089105a1856430c: Status 404 returned error can't find the container with id 72664eeba82d3a180e9c35671cd16b273e0c6a9d8d2b821e5089105a1856430c Oct 30 19:32:15.710594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.710552 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:15.710680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.710649 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:15.710721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.710696 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:15.968181 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:15.968152 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e10d4d_bb57_4275_8929_26a9d24fe937.slice/crio-d12453c3801c518221ff1462e13e95bd354f04ecdd9494d6be9b8b4c652a3294 WatchSource:0}: Error finding container d12453c3801c518221ff1462e13e95bd354f04ecdd9494d6be9b8b4c652a3294: Status 404 returned error can't find the container with id d12453c3801c518221ff1462e13e95bd354f04ecdd9494d6be9b8b4c652a3294 Oct 30 19:32:15.968880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.968844 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf"] Oct 30 19:32:15.971158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.971125 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:15.971243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.971197 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:15.971289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:15.971240 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:16.303992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.303908 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:16.464781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.438384 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:16.485302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.485266 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" event={"ID":"bf846e02-c35f-4f1c-933f-a5a8a4a64060","Type":"ContainerStarted","Data":"004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52"} Oct 30 19:32:16.485488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.485320 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" event={"ID":"bf846e02-c35f-4f1c-933f-a5a8a4a64060","Type":"ContainerStarted","Data":"72664eeba82d3a180e9c35671cd16b273e0c6a9d8d2b821e5089105a1856430c"} Oct 30 19:32:16.488393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.488369 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" event={"ID":"c0e10d4d-bb57-4275-8929-26a9d24fe937","Type":"ContainerStarted","Data":"d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd"} Oct 30 19:32:16.488497 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.488410 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" event={"ID":"c0e10d4d-bb57-4275-8929-26a9d24fe937","Type":"ContainerStarted","Data":"d12453c3801c518221ff1462e13e95bd354f04ecdd9494d6be9b8b4c652a3294"} Oct 30 19:32:16.507485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.507438 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podStartSLOduration=1.507423042 podStartE2EDuration="1.507423042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:16.506807106 +0000 UTC m=+906.464626296" watchObservedRunningTime="2026-10-30 19:32:16.507423042 +0000 UTC m=+906.465242234" Oct 30 19:32:16.532655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.532609 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podStartSLOduration=1.532595112 podStartE2EDuration="1.532595112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:16.52843087 +0000 UTC m=+906.486250061" watchObservedRunningTime="2026-10-30 19:32:16.532595112 +0000 UTC m=+906.490414298" Oct 30 19:32:16.673910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:16.673811 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:17.020023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.019844 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:17.417362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.414436 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:17.435892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.427950 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:17.435892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.428015 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:17.675611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.675533 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:17.676063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.676036 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:17.890382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:17.890351 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.051011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.048462 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.269484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.269405 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.408785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.408605 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:18.409441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.409040 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:18.674443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.674361 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:18.674725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.674686 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:18.874789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.874675 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5"] Oct 30 19:32:18.891894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.889062 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb"] Oct 30 19:32:18.981421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:18.981385 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.148686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:19.148644 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.288937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:19.287334 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.410930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:19.409613 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:19.410930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:19.409669 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:19.676373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:19.676279 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:19.676628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:19.676596 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:20.119888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:20.117059 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:32:20.365098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:20.365067 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:20.408466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:20.408382 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:20.409181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:20.409149 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:20.673588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:20.673508 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:20.673588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:20.673552 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:20.884477 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:20.883007 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:32:21.165111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:21.165079 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:21.438941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:21.417404 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:21.438941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:21.417922 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:21.674081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:21.674047 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:21.674269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:21.674115 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:22.138890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:22.137636 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.410450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:22.410358 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:22.411046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:22.410987 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:22.674712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:22.674630 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:22.674885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:22.674704 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:23.147841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.147800 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.324891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.321093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.411211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.409381 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:23.411211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.409440 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:23.674648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.674566 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:23.674648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.674621 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:23.875924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.875841 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" podUID="e693a875-05f4-44b3-a715-90736873c473" containerName="payload-processing" containerID="cri-o://1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f" gracePeriod=25 Oct 30 19:32:23.890658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:23.890466 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" podUID="8f8e509a-b187-4752-8104-98ed7b34835c" containerName="payload-pre-processing" containerID="cri-o://c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779" gracePeriod=25 Oct 30 19:32:24.052095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.051847 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:24.381083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.381059 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:32:24.408126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.405476 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:32:24.408126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.406285 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:32:24.426764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.426735 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:32:24.461774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.461740 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc\" (UniqueName: \"kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc\") pod \"e693a875-05f4-44b3-a715-90736873c473\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " Oct 30 19:32:24.461986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.461912 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume\") pod \"e693a875-05f4-44b3-a715-90736873c473\" (UID: \"e693a875-05f4-44b3-a715-90736873c473\") " Oct 30 19:32:24.462119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.462102 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd\" (UniqueName: \"kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd\") pod \"8f8e509a-b187-4752-8104-98ed7b34835c\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " Oct 30 19:32:24.462246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.462157 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume\") pod \"8f8e509a-b187-4752-8104-98ed7b34835c\" (UID: \"8f8e509a-b187-4752-8104-98ed7b34835c\") " Oct 30 19:32:24.462984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.462953 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume" pod "8f8e509a-b187-4752-8104-98ed7b34835c" (UID: "8f8e509a-b187-4752-8104-98ed7b34835c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:24.463401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.463374 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume" pod "e693a875-05f4-44b3-a715-90736873c473" (UID: "e693a875-05f4-44b3-a715-90736873c473"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:24.465163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.465140 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd" pod "8f8e509a-b187-4752-8104-98ed7b34835c" (UID: "8f8e509a-b187-4752-8104-98ed7b34835c"). InnerVolumeSpecName "kube-api-access-x7crd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:24.465311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.465268 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc" pod "e693a875-05f4-44b3-a715-90736873c473" (UID: "e693a875-05f4-44b3-a715-90736873c473"). InnerVolumeSpecName "kube-api-access-jg2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:24.565252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.565158 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2sc\" (UniqueName: \"kubernetes.io/projected/e693a875-05f4-44b3-a715-90736873c473-kube-api-access-jg2sc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.565252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.565201 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e693a875-05f4-44b3-a715-90736873c473-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.565252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.565218 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7crd\" (UniqueName: \"kubernetes.io/projected/8f8e509a-b187-4752-8104-98ed7b34835c-kube-api-access-x7crd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.565252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.565236 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8e509a-b187-4752-8104-98ed7b34835c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.673695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.673657 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:32:24.673894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.673719 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:32:24.681768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.681737 6514 generic.go:356] "Generic (PLEG): container finished" podID="8f8e509a-b187-4752-8104-98ed7b34835c" containerID="c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779" exitCode=0 Oct 30 19:32:24.681924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.681784 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" Oct 30 19:32:24.681924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.681821 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" event={"ID":"8f8e509a-b187-4752-8104-98ed7b34835c","Type":"ContainerDied","Data":"c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779"} Oct 30 19:32:24.681924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.681909 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb" event={"ID":"8f8e509a-b187-4752-8104-98ed7b34835c","Type":"ContainerDied","Data":"dc56f736944aa196604e7a8b24c5fd6dc27a5781298d7b7afcc1956dd84c1cea"} Oct 30 19:32:24.682126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.681936 6514 scope.go:122] "RemoveContainer" containerID="c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779" Oct 30 19:32:24.683708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.683681 6514 generic.go:356] "Generic (PLEG): container finished" podID="e693a875-05f4-44b3-a715-90736873c473" containerID="1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f" exitCode=0 Oct 30 19:32:24.683834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.683732 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" event={"ID":"e693a875-05f4-44b3-a715-90736873c473","Type":"ContainerDied","Data":"1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f"} Oct 30 19:32:24.683834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.683747 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" Oct 30 19:32:24.684114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.683771 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5" event={"ID":"e693a875-05f4-44b3-a715-90736873c473","Type":"ContainerDied","Data":"a1cdb789b448975da34a0547437c9362b38161319e0ca21e1e0d6ae86dbfb4d3"} Oct 30 19:32:24.694492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.694431 6514 scope.go:122] "RemoveContainer" containerID="c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779" Oct 30 19:32:24.694836 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:24.694808 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779\": container with ID starting with c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779 not found: ID does not exist" containerID="c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779" Oct 30 19:32:24.694974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.694844 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779"} err="failed to get container status \"c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779\": rpc error: code = NotFound desc = could not find container \"c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779\": container with ID starting with c3d0c36a67325828353a8d138d0159ea4dc5ec25f7e26be28b5cdad24eeea779 not found: ID does not exist" Oct 30 19:32:24.694974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.694909 6514 scope.go:122] "RemoveContainer" containerID="1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f" Oct 30 19:32:24.705373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.705285 6514 scope.go:122] "RemoveContainer" containerID="1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f" Oct 30 19:32:24.705604 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:24.705577 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f\": container with ID starting with 1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f not found: ID does not exist" containerID="1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f" Oct 30 19:32:24.705679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.705613 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f"} err="failed to get container status \"1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f\": rpc error: code = NotFound desc = could not find container \"1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f\": container with ID starting with 1723e1db6e78c7a0c8dcf6a703fbf04bb4eb854b361fda0033267afd9403f30f not found: ID does not exist" Oct 30 19:32:24.712853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.712831 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb"] Oct 30 19:32:24.716696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.716672 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5cb1c5cd-8565fccdcf-s4zvb"] Oct 30 19:32:24.727943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.727923 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5"] Oct 30 19:32:24.732590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.732565 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5cb1c5cd-9cc6f45d4-98mg5"] Oct 30 19:32:24.850146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:24.850062 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:25.770890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:25.770050 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:26.169192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.169105 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8e509a-b187-4752-8104-98ed7b34835c" path="/var/lib/kubelet/pods/8f8e509a-b187-4752-8104-98ed7b34835c/volumes" Oct 30 19:32:26.169757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.169731 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e693a875-05f4-44b3-a715-90736873c473" path="/var/lib/kubelet/pods/e693a875-05f4-44b3-a715-90736873c473/volumes" Oct 30 19:32:26.292912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.292855 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:32:26.382544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.382514 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:26.383437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.383416 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:26.385282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.385259 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:32:26.451887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.447944 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:26.451887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.448008 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:26.777663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.774598 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:26.822148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.822114 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:26.823385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.823358 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:26.828468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.828443 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4"] Oct 30 19:32:26.828801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.828768 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" podUID="c58c0949-fb4e-4b2e-ae1e-4f049df6f401" containerName="maas-api" containerID="cri-o://ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe" gracePeriod=30 Oct 30 19:32:26.832727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:26.832699 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:32:27.381478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.379479 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:27.524290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.522346 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:27.568592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.568562 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:32:27.649301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.649269 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx"] Oct 30 19:32:27.649522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.649497 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" podUID="34c5f79c-321e-49a1-9382-9f1dde2fbc37" containerName="istio-proxy" containerID="cri-o://40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6" gracePeriod=30 Oct 30 19:32:27.655331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.655300 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs\") pod \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " Oct 30 19:32:27.655458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.655357 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9\" (UniqueName: \"kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9\") pod \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " Oct 30 19:32:27.655458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.655393 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca\") pod \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " Oct 30 19:32:27.655579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.655501 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\") pod \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\" (UID: \"c58c0949-fb4e-4b2e-ae1e-4f049df6f401\") " Oct 30 19:32:27.656214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.656172 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca" pod "c58c0949-fb4e-4b2e-ae1e-4f049df6f401" (UID: "c58c0949-fb4e-4b2e-ae1e-4f049df6f401"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:27.656527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.656496 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.657052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.656998 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:27.658924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.657800 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs" pod "c58c0949-fb4e-4b2e-ae1e-4f049df6f401" (UID: "c58c0949-fb4e-4b2e-ae1e-4f049df6f401"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:27.658924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.657890 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls" pod "c58c0949-fb4e-4b2e-ae1e-4f049df6f401" (UID: "c58c0949-fb4e-4b2e-ae1e-4f049df6f401"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:27.659815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.659795 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9" pod "c58c0949-fb4e-4b2e-ae1e-4f049df6f401" (UID: "c58c0949-fb4e-4b2e-ae1e-4f049df6f401"). InnerVolumeSpecName "kube-api-access-t79m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:27.757902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.757823 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.757902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.757846 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.757902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.757857 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79m9\" (UniqueName: \"kubernetes.io/projected/c58c0949-fb4e-4b2e-ae1e-4f049df6f401-kube-api-access-t79m9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.772335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.772314 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:27.836139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.836109 6514 generic.go:356] "Generic (PLEG): container finished" podID="c58c0949-fb4e-4b2e-ae1e-4f049df6f401" containerID="ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe" exitCode=0 Oct 30 19:32:27.836565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.836156 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" Oct 30 19:32:27.836565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.836195 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" event={"ID":"c58c0949-fb4e-4b2e-ae1e-4f049df6f401","Type":"ContainerDied","Data":"ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe"} Oct 30 19:32:27.836565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.836265 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4" event={"ID":"c58c0949-fb4e-4b2e-ae1e-4f049df6f401","Type":"ContainerDied","Data":"81ac7376c2a923961a7e1752fe4bf14c822f7f2f61e8e7f5023dacfced571f9c"} Oct 30 19:32:27.836565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.836292 6514 scope.go:122] "RemoveContainer" containerID="ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe" Oct 30 19:32:27.849163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.849109 6514 scope.go:122] "RemoveContainer" containerID="ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe" Oct 30 19:32:27.849441 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:27.849413 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe\": container with ID starting with ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe not found: ID does not exist" containerID="ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe" Oct 30 19:32:27.850293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.849452 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe"} err="failed to get container status \"ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe\": rpc error: code = NotFound desc = could not find container \"ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe\": container with ID starting with ed6d75924027498babdd16f4c0284b7ef8b848ace782c0ccce8db49a6ce66fbe not found: ID does not exist" Oct 30 19:32:27.863703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.863665 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4"] Oct 30 19:32:27.867445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:27.867422 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5cb1c5cd-6cc6c5ff86-k7dc4"] Oct 30 19:32:28.325060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:28.325026 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58c0949-fb4e-4b2e-ae1e-4f049df6f401" path="/var/lib/kubelet/pods/c58c0949-fb4e-4b2e-ae1e-4f049df6f401/volumes" Oct 30 19:32:28.651483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:28.651392 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:28.767909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:28.767875 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:29.168161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.168122 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd"] Oct 30 19:32:29.170963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.170924 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e693a875-05f4-44b3-a715-90736873c473" containerName="payload-processing" Oct 30 19:32:29.170963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.170951 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e693a875-05f4-44b3-a715-90736873c473" containerName="payload-processing" Oct 30 19:32:29.171193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.170999 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8f8e509a-b187-4752-8104-98ed7b34835c" containerName="payload-pre-processing" Oct 30 19:32:29.171193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.171009 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8e509a-b187-4752-8104-98ed7b34835c" containerName="payload-pre-processing" Oct 30 19:32:29.171193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.171049 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c58c0949-fb4e-4b2e-ae1e-4f049df6f401" containerName="maas-api" Oct 30 19:32:29.171193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.171057 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58c0949-fb4e-4b2e-ae1e-4f049df6f401" containerName="maas-api" Oct 30 19:32:29.171396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.171348 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e693a875-05f4-44b3-a715-90736873c473" containerName="payload-processing" Oct 30 19:32:29.171396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.171372 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8e509a-b187-4752-8104-98ed7b34835c" containerName="payload-pre-processing" Oct 30 19:32:29.171396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.171388 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58c0949-fb4e-4b2e-ae1e-4f049df6f401" containerName="maas-api" Oct 30 19:32:29.198820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.198782 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.204585 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.204559 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-fdb355-openshift-default-dockercfg-4qqdm\"" Oct 30 19:32:29.204706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.204693 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd"] Oct 30 19:32:29.232653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.232625 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.345995 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxst\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346062 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346120 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346238 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346296 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346383 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346505 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346592 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.346885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.346707 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.369886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.368180 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.451654 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.451716 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.451799 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.451919 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.451985 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.452126 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.452171 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxst\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.452215 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.452280 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.452760 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.453484 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.453711 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.453929 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.457885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.455171 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.458784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.458366 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.485890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.462403 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.485890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.476415 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.485890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.485458 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxst\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst\") pod \"e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:29.534544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:29.534252 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:30.020750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.020597 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:30.621920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.619319 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:30.689193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.688399 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:30.689193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.688547 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:30.689193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.688640 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:30.750086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.750006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:30.786278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.786220 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48"] Oct 30 19:32:30.828287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.828259 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48"] Oct 30 19:32:30.828611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.828573 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd"] Oct 30 19:32:30.828838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.828503 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.924451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.924366 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" event={"ID":"78eb4588-c43b-4925-99c5-73c0183e10ba","Type":"ContainerStarted","Data":"c53917f3b0da1386971edab155eee287d754690e69445c799ef85a3aa8adbb96"} Oct 30 19:32:30.973796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.973741 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s76z\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.974136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.974112 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.974345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.974330 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.974502 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.974482 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.974621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.974605 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.974853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.974833 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.975146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.975123 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.975322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.975301 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:30.975510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:30.975493 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.013945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.008968 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086334 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086409 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086457 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086527 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086601 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s76z\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086663 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086759 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086805 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.086884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.086828 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.109883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.106313 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.109883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.106566 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.109883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.109784 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.110313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.110284 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.110617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.110575 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.139052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.136566 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.149401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.149345 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.196883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.186292 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s76z\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.228963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.228930 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5cd48\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:31.292016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:31.282105 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:32.048759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.048718 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" event={"ID":"78eb4588-c43b-4925-99c5-73c0183e10ba","Type":"ContainerStarted","Data":"a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9"} Oct 30 19:32:32.087065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.084826 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podStartSLOduration=3.084808397 podStartE2EDuration="3.084808397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:32.081504028 +0000 UTC m=+922.039323217" watchObservedRunningTime="2026-10-30 19:32:32.084808397 +0000 UTC m=+922.042627587" Oct 30 19:32:32.396932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.316775 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:32.535558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.535457 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:32.547328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.546422 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:32.583668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.583624 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:32.583773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.583728 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:32.583827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.583778 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:32.591549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:32.591527 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48"] Oct 30 19:32:33.031755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.031692 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv"] Oct 30 19:32:33.035412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.035389 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:32:33.036750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.036728 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23af4310-qvjpv"] Oct 30 19:32:33.059556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.059525 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" event={"ID":"263ef9f8-23c6-4bee-ae1f-2dffc850377e","Type":"ContainerStarted","Data":"515383a8fa0b6062b5456c1bc1092c8b33d891c5491d6b3acf355708540f77a7"} Oct 30 19:32:33.060227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.059586 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" event={"ID":"263ef9f8-23c6-4bee-ae1f-2dffc850377e","Type":"ContainerStarted","Data":"c933a2f9a184c5d3cd80ddcc6441ccc7c29241639b4d1405c388e112e28117eb"} Oct 30 19:32:33.061550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.061520 6514 generic.go:356] "Generic (PLEG): container finished" podID="34c5f79c-321e-49a1-9382-9f1dde2fbc37" containerID="40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6" exitCode=0 Oct 30 19:32:33.061665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.061561 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" Oct 30 19:32:33.061768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.061740 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" event={"ID":"34c5f79c-321e-49a1-9382-9f1dde2fbc37","Type":"ContainerDied","Data":"40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6"} Oct 30 19:32:33.061966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.061948 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx" event={"ID":"34c5f79c-321e-49a1-9382-9f1dde2fbc37","Type":"ContainerDied","Data":"adbb8766fa5cf94f776a3801b4e121fc3c6dc92233cec27534b6e9d22f6ebb95"} Oct 30 19:32:33.062105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.062083 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:33.062345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.062125 6514 scope.go:122] "RemoveContainer" containerID="40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6" Oct 30 19:32:33.064013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.063912 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:32:33.075419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.075401 6514 scope.go:122] "RemoveContainer" containerID="40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6" Oct 30 19:32:33.075736 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:33.075709 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6\": container with ID starting with 40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6 not found: ID does not exist" containerID="40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6" Oct 30 19:32:33.075822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.075742 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6"} err="failed to get container status \"40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6\": rpc error: code = NotFound desc = could not find container \"40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6\": container with ID starting with 40a10803d9e4bb227579a1f09e5b1a8281adbaed1c8dc1c634950ec07bd180b6 not found: ID does not exist" Oct 30 19:32:33.086965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.086922 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podStartSLOduration=3.086909347 podStartE2EDuration="3.086909347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:33.083503632 +0000 UTC m=+923.041322827" watchObservedRunningTime="2026-10-30 19:32:33.086909347 +0000 UTC m=+923.044728535" Oct 30 19:32:33.155442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.155417 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.155730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.155707 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.155967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.155948 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.156098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156082 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.156235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156219 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.156376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156102 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:33.156508 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156493 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.156616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156596 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:33.156684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156614 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:33.156858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.156839 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.158200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.157713 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:33.158416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.158397 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.158633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.158617 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs\") pod \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\" (UID: \"34c5f79c-321e-49a1-9382-9f1dde2fbc37\") " Oct 30 19:32:33.158966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.158938 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:33.159847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.159823 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.159983 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.159967 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.160077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.160064 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.160162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.160150 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.160311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.160265 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.169073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.169044 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "kube-api-access-5jz46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:33.174084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.174057 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:33.175444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.175414 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:33.176029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.176006 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy" pod "34c5f79c-321e-49a1-9382-9f1dde2fbc37" (UID: "34c5f79c-321e-49a1-9382-9f1dde2fbc37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:33.262362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.262292 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.262362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.262318 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.262362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.262329 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.262362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.262339 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jz46\" (UniqueName: \"kubernetes.io/projected/34c5f79c-321e-49a1-9382-9f1dde2fbc37-kube-api-access-5jz46\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:33.292942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.292909 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:33.294494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.294464 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:33.294614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.294539 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:33.435784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.435718 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx"] Oct 30 19:32:33.441480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.441437 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5cb1c5cd-openshift-default-67897f4645-w4hvx"] Oct 30 19:32:33.478834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:33.478804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.161122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.161085 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c5f79c-321e-49a1-9382-9f1dde2fbc37" path="/var/lib/kubelet/pods/34c5f79c-321e-49a1-9382-9f1dde2fbc37/volumes" Oct 30 19:32:34.161614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.161598 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c05307-98d1-4ca4-8b84-28ca13dd3f3d" path="/var/lib/kubelet/pods/91c05307-98d1-4ca4-8b84-28ca13dd3f3d/volumes" Oct 30 19:32:34.316023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.315979 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:34.316225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.316049 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:34.361384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.361352 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.538888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.538760 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.722939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.722893 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.838080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:34.837991 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:35.294178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.294146 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:35.294710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.294676 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:35.388477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.388439 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7162-4d2823"] Oct 30 19:32:35.389266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.389246 6514 cpu_manager.go:414] "containerMap: removing container" podUID="34c5f79c-321e-49a1-9382-9f1dde2fbc37" containerName="istio-proxy" Oct 30 19:32:35.389318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.389268 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c5f79c-321e-49a1-9382-9f1dde2fbc37" containerName="istio-proxy" Oct 30 19:32:35.389414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.389404 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c5f79c-321e-49a1-9382-9f1dde2fbc37" containerName="istio-proxy" Oct 30 19:32:35.409886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.395311 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:35.413611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.413056 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvcnk\"" Oct 30 19:32:35.434626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.434595 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-4d2823"] Oct 30 19:32:35.439900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.439832 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlcg\" (UniqueName: \"kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg\") pod \"test-curl-7162-4d2823\" (UID: \"96adb344-aba8-4898-9057-6db2e10659f8\") " pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:35.456995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.456964 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:35.590141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.573056 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlcg\" (UniqueName: \"kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg\") pod \"test-curl-7162-4d2823\" (UID: \"96adb344-aba8-4898-9057-6db2e10659f8\") " pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:35.652054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.651792 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlcg\" (UniqueName: \"kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg\") pod \"test-curl-7162-4d2823\" (UID: \"96adb344-aba8-4898-9057-6db2e10659f8\") " pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:35.734484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:35.734452 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:36.028252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.028207 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:36.029247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.029219 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4"] Oct 30 19:32:36.034746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.034721 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.044076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.044049 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fdb355\"" Oct 30 19:32:36.048398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.048375 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4"] Oct 30 19:32:36.052471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.052263 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-4d2823"] Oct 30 19:32:36.057935 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:36.057910 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96adb344_aba8_4898_9057_6db2e10659f8.slice/crio-50cdb76045538300cdedf0764b8607ecb053a7c9cefac8c12ab41d039533d717 WatchSource:0}: Error finding container 50cdb76045538300cdedf0764b8607ecb053a7c9cefac8c12ab41d039533d717: Status 404 returned error can't find the container with id 50cdb76045538300cdedf0764b8607ecb053a7c9cefac8c12ab41d039533d717 Oct 30 19:32:36.081082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.081051 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgslp\" (UniqueName: \"kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp\") pod \"payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.081362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.081340 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.135538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.135509 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc"] Oct 30 19:32:36.178602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.178502 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:36.318546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.318441 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.319452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.319408 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgslp\" (UniqueName: \"kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp\") pod \"payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.321812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.321787 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.369687 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.369601 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgslp\" (UniqueName: \"kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp\") pod \"payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.374312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.371470 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:36.374312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.371535 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:36.383723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.383460 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:36.667295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.667270 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc"] Oct 30 19:32:36.667528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.667485 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-4d2823" event={"ID":"96adb344-aba8-4898-9057-6db2e10659f8","Type":"ContainerStarted","Data":"50cdb76045538300cdedf0764b8607ecb053a7c9cefac8c12ab41d039533d717"} Oct 30 19:32:36.672993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.669306 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:36.675918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.675729 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fdb355-dockercfg-9gjl7\"" Oct 30 19:32:36.822386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.822344 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tw4v\" (UniqueName: \"kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v\") pod \"payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:36.822576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.822509 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:36.826757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.826729 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:36.989889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.951894 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tw4v\" (UniqueName: \"kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v\") pod \"payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:36.989889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.952013 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:36.989889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.953369 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:36.989889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:36.976896 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tw4v\" (UniqueName: \"kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v\") pod \"payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:37.023231 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.023197 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:37.049837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.049681 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4"] Oct 30 19:32:37.150379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.150338 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-4d2823" event={"ID":"96adb344-aba8-4898-9057-6db2e10659f8","Type":"ContainerStarted","Data":"4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7"} Oct 30 19:32:37.153506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.153484 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" event={"ID":"a8463f9f-e220-4d14-835c-2ae558cb209a","Type":"ContainerStarted","Data":"86dd211b55089434b746875b45b395202a375749f91804f92aed83b3d9def4eb"} Oct 30 19:32:37.174403 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.174359 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7162-4d2823" podStartSLOduration=2.174345104 podStartE2EDuration="2.174345104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:37.173301876 +0000 UTC m=+927.131121064" watchObservedRunningTime="2026-10-30 19:32:37.174345104 +0000 UTC m=+927.132164292" Oct 30 19:32:37.219460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.219368 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:37.314393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.313924 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:37.314393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.313993 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:37.346255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.346190 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:37.383858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.383830 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc"] Oct 30 19:32:37.384257 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:37.384218 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9968f0_efba_4537_8f96_359415fa147f.slice/crio-63834c589c47018dc4ce3ff2e00d983cf6103ff9651d8e2dc23539a0a2fa9ed9 WatchSource:0}: Error finding container 63834c589c47018dc4ce3ff2e00d983cf6103ff9651d8e2dc23539a0a2fa9ed9: Status 404 returned error can't find the container with id 63834c589c47018dc4ce3ff2e00d983cf6103ff9651d8e2dc23539a0a2fa9ed9 Oct 30 19:32:37.817830 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.817799 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:37.975566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.975491 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7162-4d2823"] Oct 30 19:32:37.981921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:37.979128 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7162-4d2823"] Oct 30 19:32:38.124359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.124278 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7162-f76103"] Oct 30 19:32:38.125158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.125138 6514 cpu_manager.go:414] "containerMap: removing container" podUID="96adb344-aba8-4898-9057-6db2e10659f8" containerName="test-curl-7162-4d2823" Oct 30 19:32:38.125158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.125159 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96adb344-aba8-4898-9057-6db2e10659f8" containerName="test-curl-7162-4d2823" Oct 30 19:32:38.125412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.125397 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="96adb344-aba8-4898-9057-6db2e10659f8" containerName="test-curl-7162-4d2823" Oct 30 19:32:38.129302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.129280 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:38.133458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.133427 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-f76103"] Oct 30 19:32:38.166910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.166846 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" event={"ID":"a8463f9f-e220-4d14-835c-2ae558cb209a","Type":"ContainerStarted","Data":"e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6"} Oct 30 19:32:38.167779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.167760 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:38.169753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.169728 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" event={"ID":"4d9968f0-efba-4537-8f96-359415fa147f","Type":"ContainerStarted","Data":"f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc"} Oct 30 19:32:38.169834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.169772 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" event={"ID":"4d9968f0-efba-4537-8f96-359415fa147f","Type":"ContainerStarted","Data":"63834c589c47018dc4ce3ff2e00d983cf6103ff9651d8e2dc23539a0a2fa9ed9"} Oct 30 19:32:38.188969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.188930 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" podStartSLOduration=2.188917702 podStartE2EDuration="2.188917702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:38.184604023 +0000 UTC m=+928.142423210" watchObservedRunningTime="2026-10-30 19:32:38.188917702 +0000 UTC m=+928.146736889" Oct 30 19:32:38.219321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.219267 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" podStartSLOduration=2.219249655 podStartE2EDuration="2.219249655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:38.213142507 +0000 UTC m=+928.170961697" watchObservedRunningTime="2026-10-30 19:32:38.219249655 +0000 UTC m=+928.177068846" Oct 30 19:32:38.279782 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.279652 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pkr\" (UniqueName: \"kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr\") pod \"test-curl-7162-f76103\" (UID: \"5783edb7-53c3-4390-92df-e8ad1d178d99\") " pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:38.311884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.298161 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:38.311884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.298212 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:38.383110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.382997 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pkr\" (UniqueName: \"kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr\") pod \"test-curl-7162-f76103\" (UID: \"5783edb7-53c3-4390-92df-e8ad1d178d99\") " pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:38.441938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.433651 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pkr\" (UniqueName: \"kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr\") pod \"test-curl-7162-f76103\" (UID: \"5783edb7-53c3-4390-92df-e8ad1d178d99\") " pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:38.454449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.453223 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:38.732314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.732284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:38.871765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.871731 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-f76103"] Oct 30 19:32:38.889185 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:38.889138 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5783edb7_53c3_4390_92df_e8ad1d178d99.slice/crio-d9a7b22667cf602d19b1864df995258656597991d4a0965a9f6286820dc2c7db WatchSource:0}: Error finding container d9a7b22667cf602d19b1864df995258656597991d4a0965a9f6286820dc2c7db: Status 404 returned error can't find the container with id d9a7b22667cf602d19b1864df995258656597991d4a0965a9f6286820dc2c7db Oct 30 19:32:38.991827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:38.991741 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.178590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.178558 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-f76103" event={"ID":"5783edb7-53c3-4390-92df-e8ad1d178d99","Type":"ContainerStarted","Data":"9cd2af2784d83320fe5dc302b1d057800b4bc3567bc6b2b1455d519ad86a65be"} Oct 30 19:32:39.178772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.178601 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-f76103" event={"ID":"5783edb7-53c3-4390-92df-e8ad1d178d99","Type":"ContainerStarted","Data":"d9a7b22667cf602d19b1864df995258656597991d4a0965a9f6286820dc2c7db"} Oct 30 19:32:39.178772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.178674 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7162-4d2823" podUID="96adb344-aba8-4898-9057-6db2e10659f8" containerName="test-curl-7162-4d2823" containerID="cri-o://4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7" gracePeriod=2 Oct 30 19:32:39.178932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.178800 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:39.204834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.204781 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7162-f76103" podStartSLOduration=1.204765013 podStartE2EDuration="1.204765013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:39.202459201 +0000 UTC m=+929.160278371" watchObservedRunningTime="2026-10-30 19:32:39.204765013 +0000 UTC m=+929.162584202" Oct 30 19:32:39.293290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.293205 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:39.293290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.293259 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:39.681383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.681333 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.935493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.935418 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7162-f76103"] Oct 30 19:32:39.942554 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:39.942515 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7162-f76103"] Oct 30 19:32:40.088829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.088797 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7162-9a9a23"] Oct 30 19:32:40.089653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.089635 6514 cpu_manager.go:414] "containerMap: removing container" podUID="5783edb7-53c3-4390-92df-e8ad1d178d99" containerName="test-curl-7162-f76103" Oct 30 19:32:40.089718 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.089655 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5783edb7-53c3-4390-92df-e8ad1d178d99" containerName="test-curl-7162-f76103" Oct 30 19:32:40.089763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.089753 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="5783edb7-53c3-4390-92df-e8ad1d178d99" containerName="test-curl-7162-f76103" Oct 30 19:32:40.094452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.094425 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:40.101411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.101383 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-9a9a23"] Oct 30 19:32:40.254611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.254520 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4qq\" (UniqueName: \"kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq\") pod \"test-curl-7162-9a9a23\" (UID: \"84518fbb-0638-471c-b970-a1f613e92063\") " pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:40.293237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.293202 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5cd48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:32:40.293426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.293267 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 19:32:40.356120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.356090 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4qq\" (UniqueName: \"kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq\") pod \"test-curl-7162-9a9a23\" (UID: \"84518fbb-0638-471c-b970-a1f613e92063\") " pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:40.368147 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.368116 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4qq\" (UniqueName: \"kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq\") pod \"test-curl-7162-9a9a23\" (UID: \"84518fbb-0638-471c-b970-a1f613e92063\") " pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:40.411700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.411657 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:40.597857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.597775 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:40.742280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:40.742246 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-9a9a23"] Oct 30 19:32:40.774485 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:40.774431 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84518fbb_0638_471c_b970_a1f613e92063.slice/crio-f7f9d0fa6b6015cff1f6558999dc9f31b473b6e2b33d36924e50a57bc074a0a5 WatchSource:0}: Error finding container f7f9d0fa6b6015cff1f6558999dc9f31b473b6e2b33d36924e50a57bc074a0a5: Status 404 returned error can't find the container with id f7f9d0fa6b6015cff1f6558999dc9f31b473b6e2b33d36924e50a57bc074a0a5 Oct 30 19:32:41.200729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.200595 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-9a9a23" event={"ID":"84518fbb-0638-471c-b970-a1f613e92063","Type":"ContainerStarted","Data":"dc6a86b492057c8d494fa9920e4733adba3165da4ad5f7e94e0812547db01035"} Oct 30 19:32:41.200729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.200654 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-9a9a23" event={"ID":"84518fbb-0638-471c-b970-a1f613e92063","Type":"ContainerStarted","Data":"f7f9d0fa6b6015cff1f6558999dc9f31b473b6e2b33d36924e50a57bc074a0a5"} Oct 30 19:32:41.201028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.200945 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7162-f76103" podUID="5783edb7-53c3-4390-92df-e8ad1d178d99" containerName="test-curl-7162-f76103" containerID="cri-o://9cd2af2784d83320fe5dc302b1d057800b4bc3567bc6b2b1455d519ad86a65be" gracePeriod=2 Oct 30 19:32:41.222504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.222326 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7162-9a9a23" podStartSLOduration=1.222308484 podStartE2EDuration="1.222308484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:41.218340651 +0000 UTC m=+931.176159841" watchObservedRunningTime="2026-10-30 19:32:41.222308484 +0000 UTC m=+931.180127674" Oct 30 19:32:41.349964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.349930 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:41.396977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.383992 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:41.415359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.412884 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg\" (UniqueName: \"kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg\") pod \"96adb344-aba8-4898-9057-6db2e10659f8\" (UID: \"96adb344-aba8-4898-9057-6db2e10659f8\") " Oct 30 19:32:41.474405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.474285 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg" pod "96adb344-aba8-4898-9057-6db2e10659f8" (UID: "96adb344-aba8-4898-9057-6db2e10659f8"). InnerVolumeSpecName "kube-api-access-5nlcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:41.572900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.572848 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlcg\" (UniqueName: \"kubernetes.io/projected/96adb344-aba8-4898-9057-6db2e10659f8-kube-api-access-5nlcg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:41.818444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.818360 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:41.818935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.818908 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:41.819817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.819788 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:32:41.966911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.966857 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7162-9a9a23"] Oct 30 19:32:41.971652 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:41.971622 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7162-9a9a23"] Oct 30 19:32:42.129504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.129421 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7162-06ce6b"] Oct 30 19:32:42.130750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.130727 6514 cpu_manager.go:414] "containerMap: removing container" podUID="84518fbb-0638-471c-b970-a1f613e92063" containerName="test-curl-7162-9a9a23" Oct 30 19:32:42.130826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.130752 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84518fbb-0638-471c-b970-a1f613e92063" containerName="test-curl-7162-9a9a23" Oct 30 19:32:42.131071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.131054 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="84518fbb-0638-471c-b970-a1f613e92063" containerName="test-curl-7162-9a9a23" Oct 30 19:32:42.136207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.136180 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:42.138133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.138108 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-06ce6b"] Oct 30 19:32:42.143407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.143378 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:42.184895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.172314 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96adb344-aba8-4898-9057-6db2e10659f8" path="/var/lib/kubelet/pods/96adb344-aba8-4898-9057-6db2e10659f8/volumes" Oct 30 19:32:42.303970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.302531 6514 generic.go:356] "Generic (PLEG): container finished" podID="96adb344-aba8-4898-9057-6db2e10659f8" containerID="4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7" exitCode=137 Oct 30 19:32:42.303970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.303639 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-4d2823" Oct 30 19:32:42.307614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.307571 6514 scope.go:122] "RemoveContainer" containerID="4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7" Oct 30 19:32:42.319064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.315838 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsjb\" (UniqueName: \"kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb\") pod \"test-curl-7162-06ce6b\" (UID: \"f799f44f-3fe4-431a-838e-7e5a56cd73c3\") " pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:42.339580 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.339552 6514 scope.go:122] "RemoveContainer" containerID="4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7" Oct 30 19:32:42.339973 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:42.339950 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7\": container with ID starting with 4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7 not found: ID does not exist" containerID="4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7" Oct 30 19:32:42.340063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.339984 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7"} err="failed to get container status \"4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7\": rpc error: code = NotFound desc = could not find container \"4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7\": container with ID starting with 4f8cd186e09ec24cccb114b24c2fafeb0bc2ae9dfdd34262601b8ee2c64907b7 not found: ID does not exist" Oct 30 19:32:42.418074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.418003 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsjb\" (UniqueName: \"kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb\") pod \"test-curl-7162-06ce6b\" (UID: \"f799f44f-3fe4-431a-838e-7e5a56cd73c3\") " pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:42.427275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.427246 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsjb\" (UniqueName: \"kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb\") pod \"test-curl-7162-06ce6b\" (UID: \"f799f44f-3fe4-431a-838e-7e5a56cd73c3\") " pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:42.467967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.467941 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:42.758698 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:42.758621 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf799f44f_3fe4_431a_838e_7e5a56cd73c3.slice/crio-94184846d4f40c87bd7327f4bf75bbc0ef629fd4e855a09769d26cef63941587 WatchSource:0}: Error finding container 94184846d4f40c87bd7327f4bf75bbc0ef629fd4e855a09769d26cef63941587: Status 404 returned error can't find the container with id 94184846d4f40c87bd7327f4bf75bbc0ef629fd4e855a09769d26cef63941587 Oct 30 19:32:42.763773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.763749 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-06ce6b"] Oct 30 19:32:42.940413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:42.940381 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.314106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.314042 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-06ce6b" event={"ID":"f799f44f-3fe4-431a-838e-7e5a56cd73c3","Type":"ContainerStarted","Data":"1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e"} Oct 30 19:32:43.314106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.314093 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-06ce6b" event={"ID":"f799f44f-3fe4-431a-838e-7e5a56cd73c3","Type":"ContainerStarted","Data":"94184846d4f40c87bd7327f4bf75bbc0ef629fd4e855a09769d26cef63941587"} Oct 30 19:32:43.318417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.318387 6514 generic.go:356] "Generic (PLEG): container finished" podID="5783edb7-53c3-4390-92df-e8ad1d178d99" containerID="9cd2af2784d83320fe5dc302b1d057800b4bc3567bc6b2b1455d519ad86a65be" exitCode=137 Oct 30 19:32:43.319537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.319507 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7162-9a9a23" podUID="84518fbb-0638-471c-b970-a1f613e92063" containerName="test-curl-7162-9a9a23" containerID="cri-o://dc6a86b492057c8d494fa9920e4733adba3165da4ad5f7e94e0812547db01035" gracePeriod=2 Oct 30 19:32:43.332832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.332782 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7162-06ce6b" podStartSLOduration=1.332766004 podStartE2EDuration="1.332766004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:43.328597802 +0000 UTC m=+933.286416991" watchObservedRunningTime="2026-10-30 19:32:43.332766004 +0000 UTC m=+933.290585194" Oct 30 19:32:43.388487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.388461 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:43.545549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.544753 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr\" (UniqueName: \"kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr\") pod \"5783edb7-53c3-4390-92df-e8ad1d178d99\" (UID: \"5783edb7-53c3-4390-92df-e8ad1d178d99\") " Oct 30 19:32:43.556922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.556884 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr" pod "5783edb7-53c3-4390-92df-e8ad1d178d99" (UID: "5783edb7-53c3-4390-92df-e8ad1d178d99"). InnerVolumeSpecName "kube-api-access-d9pkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:43.583163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.583135 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm"] Oct 30 19:32:43.603067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.603038 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm"] Oct 30 19:32:43.603239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.603141 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.607333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.607310 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-fdb355-serving-cert\"" Oct 30 19:32:43.649365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.649336 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pkr\" (UniqueName: \"kubernetes.io/projected/5783edb7-53c3-4390-92df-e8ad1d178d99-kube-api-access-d9pkr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:43.756955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.756414 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.756955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.756498 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.756955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.756576 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcgk\" (UniqueName: \"kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.756955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.756656 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.768887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.767016 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.868565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.868423 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.868812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.868793 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.869972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.869944 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcgk\" (UniqueName: \"kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.870214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.870188 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.870533 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:43.869007 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-serving-cert: secret "maas-api-e2e-shared-a-w3-fdb355-serving-cert" not found Oct 30 19:32:43.870719 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:43.870705 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls podName:c5c6654f-adc5-4091-a7b2-40af102bb695 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:44.370681299 +0000 UTC m=+934.328500468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls") pod "maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" (UID: "c5c6654f-adc5-4091-a7b2-40af102bb695") : secret "maas-api-e2e-shared-a-w3-fdb355-serving-cert" not found Oct 30 19:32:43.870836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.868532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.871010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.870823 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.871401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.871373 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:43.883752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:43.883729 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcgk\" (UniqueName: \"kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:44.064726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.064696 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:44.086952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.086914 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7162-06ce6b"] Oct 30 19:32:44.093249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.093221 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7162-06ce6b"] Oct 30 19:32:44.163696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.163625 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5783edb7-53c3-4390-92df-e8ad1d178d99" path="/var/lib/kubelet/pods/5783edb7-53c3-4390-92df-e8ad1d178d99/volumes" Oct 30 19:32:44.396034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.384398 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:44.428938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.421737 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:44.446939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.442380 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-f76103" Oct 30 19:32:44.446939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.443032 6514 scope.go:122] "RemoveContainer" containerID="9cd2af2784d83320fe5dc302b1d057800b4bc3567bc6b2b1455d519ad86a65be" Oct 30 19:32:44.523112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.522759 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:44.737882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.737786 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:44.879367 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:32:44.870230 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c6654f_adc5_4091_a7b2_40af102bb695.slice/crio-f2ea25fe111da0538c98dbc4b39a49c8483cf8701c2dc868194b64fd842a64c5 WatchSource:0}: Error finding container f2ea25fe111da0538c98dbc4b39a49c8483cf8701c2dc868194b64fd842a64c5: Status 404 returned error can't find the container with id f2ea25fe111da0538c98dbc4b39a49c8483cf8701c2dc868194b64fd842a64c5 Oct 30 19:32:44.879367 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:44.873793 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm"] Oct 30 19:32:45.079137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.079062 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:45.487499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.487463 6514 generic.go:356] "Generic (PLEG): container finished" podID="84518fbb-0638-471c-b970-a1f613e92063" containerID="dc6a86b492057c8d494fa9920e4733adba3165da4ad5f7e94e0812547db01035" exitCode=137 Oct 30 19:32:45.489519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.489487 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7162-06ce6b" podUID="f799f44f-3fe4-431a-838e-7e5a56cd73c3" containerName="test-curl-7162-06ce6b" containerID="cri-o://1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e" gracePeriod=2 Oct 30 19:32:45.489847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.489820 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" event={"ID":"c5c6654f-adc5-4091-a7b2-40af102bb695","Type":"ContainerStarted","Data":"f2ea25fe111da0538c98dbc4b39a49c8483cf8701c2dc868194b64fd842a64c5"} Oct 30 19:32:45.527966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.527944 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:45.542272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.542247 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:45.665021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.663744 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq\" (UniqueName: \"kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq\") pod \"84518fbb-0638-471c-b970-a1f613e92063\" (UID: \"84518fbb-0638-471c-b970-a1f613e92063\") " Oct 30 19:32:45.751888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.748432 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:45.813186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.813139 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq" pod "84518fbb-0638-471c-b970-a1f613e92063" (UID: "84518fbb-0638-471c-b970-a1f613e92063"). InnerVolumeSpecName "kube-api-access-tj4qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:45.835910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:45.824967 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4qq\" (UniqueName: \"kubernetes.io/projected/84518fbb-0638-471c-b970-a1f613e92063-kube-api-access-tj4qq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:46.162821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.162790 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84518fbb-0638-471c-b970-a1f613e92063" path="/var/lib/kubelet/pods/84518fbb-0638-471c-b970-a1f613e92063/volumes" Oct 30 19:32:46.326689 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.326657 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:46.515120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.515079 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" event={"ID":"c5c6654f-adc5-4091-a7b2-40af102bb695","Type":"ContainerStarted","Data":"2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24"} Oct 30 19:32:46.515346 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.515301 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:46.516742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.516715 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-9a9a23" Oct 30 19:32:46.516907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.516719 6514 scope.go:122] "RemoveContainer" containerID="dc6a86b492057c8d494fa9920e4733adba3165da4ad5f7e94e0812547db01035" Oct 30 19:32:46.540961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.540917 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" podStartSLOduration=3.034157076 podStartE2EDuration="3.540902885s" totalImagesPullingTime="506.745809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:46.535449008 +0000 UTC m=+936.493268195" watchObservedRunningTime="2026-10-30 19:32:46.540902885 +0000 UTC m=+936.498722073" Oct 30 19:32:46.676058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:46.676029 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:47.506906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:47.506427 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:47.782648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:47.782622 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:47.908759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:47.908727 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb\" (UniqueName: \"kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb\") pod \"f799f44f-3fe4-431a-838e-7e5a56cd73c3\" (UID: \"f799f44f-3fe4-431a-838e-7e5a56cd73c3\") " Oct 30 19:32:47.911250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:47.911224 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb" pod "f799f44f-3fe4-431a-838e-7e5a56cd73c3" (UID: "f799f44f-3fe4-431a-838e-7e5a56cd73c3"). InnerVolumeSpecName "kube-api-access-zbsjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:48.011050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.010975 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbsjb\" (UniqueName: \"kubernetes.io/projected/f799f44f-3fe4-431a-838e-7e5a56cd73c3-kube-api-access-zbsjb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:48.021764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.021737 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:48.197887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.197342 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f799f44f-3fe4-431a-838e-7e5a56cd73c3" path="/var/lib/kubelet/pods/f799f44f-3fe4-431a-838e-7e5a56cd73c3/volumes" Oct 30 19:32:48.588729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.588694 6514 generic.go:356] "Generic (PLEG): container finished" podID="f799f44f-3fe4-431a-838e-7e5a56cd73c3" containerID="1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e" exitCode=137 Oct 30 19:32:48.589225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.588743 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-06ce6b" Oct 30 19:32:48.589225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.588800 6514 scope.go:122] "RemoveContainer" containerID="1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e" Oct 30 19:32:48.601743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.601727 6514 scope.go:122] "RemoveContainer" containerID="1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e" Oct 30 19:32:48.602044 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:48.602026 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e\": container with ID starting with 1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e not found: ID does not exist" containerID="1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e" Oct 30 19:32:48.602119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.602050 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e"} err="failed to get container status \"1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e\": rpc error: code = NotFound desc = could not find container \"1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e\": container with ID starting with 1863e28d15e0fca596960f2156bae48c3a3eced5d6c6565bf230a26615d70b4e not found: ID does not exist" Oct 30 19:32:48.639585 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:48.639563 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.534891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:49.527294 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:50.189734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:50.189707 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:32:50.190213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:50.189848 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:32:50.437715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:50.437671 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:50.658937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:50.648139 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:50.919676 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:32:50.919565 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:32:51.910338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:51.910305 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:52.067122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:52.067089 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:52.635887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:52.635480 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:32:53.486073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:53.486040 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:54.395473 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:54.395440 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:55.347889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:55.341058 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:55.687590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:55.680940 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:56.042940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:56.027472 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:56.093888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:56.078945 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:56.162836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:56.162801 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:57.326898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:57.319720 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:58.498236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:58.498200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:59.392889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:59.388441 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:59.838514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:32:59.838478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:00.352828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:00.352787 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:00.498076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:00.498041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:00.786630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:00.786591 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:01.396891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:01.394906 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:01.637282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:01.630156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:02.501558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:02.501517 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:03.269250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.269217 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g"] Oct 30 19:33:03.270120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.270093 6514 cpu_manager.go:414] "containerMap: removing container" podUID="f799f44f-3fe4-431a-838e-7e5a56cd73c3" containerName="test-curl-7162-06ce6b" Oct 30 19:33:03.270120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.270115 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f799f44f-3fe4-431a-838e-7e5a56cd73c3" containerName="test-curl-7162-06ce6b" Oct 30 19:33:03.270365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.270350 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="f799f44f-3fe4-431a-838e-7e5a56cd73c3" containerName="test-curl-7162-06ce6b" Oct 30 19:33:03.283136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.283109 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.292135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.292104 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g"] Oct 30 19:33:03.292396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.292372 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-20e66fa4-serving-cert\"" Oct 30 19:33:03.298691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.298661 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.299192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.298938 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.299192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.298985 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.299192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.299169 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lqq\" (UniqueName: \"kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.349168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.349115 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:03.400931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.400849 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lqq\" (UniqueName: \"kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.401143 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.401001 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.401230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.401172 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.401230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.401204 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.401907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.401878 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.403545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.403521 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.403630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.403595 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.409685 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.409664 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lqq\" (UniqueName: \"kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.497680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.497645 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:03.615934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.611574 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:03.962533 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:33:03.962492 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4cbe2f_ecea_499d_b5ff_9fbc06ffc94a.slice/crio-32313c2c946c7fcf4254913afcbb6629406c4b881e24dcdc84a2199b04da1316 WatchSource:0}: Error finding container 32313c2c946c7fcf4254913afcbb6629406c4b881e24dcdc84a2199b04da1316: Status 404 returned error can't find the container with id 32313c2c946c7fcf4254913afcbb6629406c4b881e24dcdc84a2199b04da1316 Oct 30 19:33:03.962927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:03.962904 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g"] Oct 30 19:33:04.618533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:04.618499 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:04.977645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:04.977612 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" event={"ID":"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a","Type":"ContainerStarted","Data":"21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439"} Oct 30 19:33:04.977801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:04.977666 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" event={"ID":"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a","Type":"ContainerStarted","Data":"32313c2c946c7fcf4254913afcbb6629406c4b881e24dcdc84a2199b04da1316"} Oct 30 19:33:04.977801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:04.977697 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:05.000341 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:05.000292 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" podStartSLOduration=1.498092758 podStartE2EDuration="2.000278109s" totalImagesPullingTime="502.185351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:04.996342345 +0000 UTC m=+954.954161533" watchObservedRunningTime="2026-10-30 19:33:05.000278109 +0000 UTC m=+954.958097298" Oct 30 19:33:05.754801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:05.754749 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:05.949900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:05.947589 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:06.976126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:06.976051 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.053739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.053706 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n"] Oct 30 19:33:07.058779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.058760 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.071163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.071137 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n"] Oct 30 19:33:07.109739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.109710 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.166889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.163240 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-976b80-dockercfg-nc6g9\"" Oct 30 19:33:07.166889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.163349 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-976b80\"" Oct 30 19:33:07.269331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.267211 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt475\" (UniqueName: \"kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475\") pod \"payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.269331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.267556 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.369779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.369740 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt475\" (UniqueName: \"kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475\") pod \"payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.372709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.372677 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.373998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.373973 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.444887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.444649 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt475\" (UniqueName: \"kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475\") pod \"payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.676223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.676149 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:07.966316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.966290 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.969709 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:33:07.969676 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2552fab8_b507_4c24_b3a4_fd1e06c51835.slice/crio-424851b18c25eb09e0c3e6694791288b5c82b5b98f1bcdc88eff2f92fec6b281 WatchSource:0}: Error finding container 424851b18c25eb09e0c3e6694791288b5c82b5b98f1bcdc88eff2f92fec6b281: Status 404 returned error can't find the container with id 424851b18c25eb09e0c3e6694791288b5c82b5b98f1bcdc88eff2f92fec6b281 Oct 30 19:33:07.970053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:07.970031 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n"] Oct 30 19:33:08.124234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:08.124199 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" event={"ID":"2552fab8-b507-4c24-b3a4-fd1e06c51835","Type":"ContainerStarted","Data":"424851b18c25eb09e0c3e6694791288b5c82b5b98f1bcdc88eff2f92fec6b281"} Oct 30 19:33:08.791566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:08.791533 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:09.133735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:09.133638 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" event={"ID":"2552fab8-b507-4c24-b3a4-fd1e06c51835","Type":"ContainerStarted","Data":"39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912"} Oct 30 19:33:09.134120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:09.133741 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:09.162319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:09.162260 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" podStartSLOduration=2.162238881 podStartE2EDuration="2.162238881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:09.158248359 +0000 UTC m=+959.116067559" watchObservedRunningTime="2026-10-30 19:33:09.162238881 +0000 UTC m=+959.120058071" Oct 30 19:33:09.739593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:09.739556 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:10.439309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:10.439281 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:11.085532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:11.085494 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:33:11.137121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:11.137093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:11.353943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:11.348545 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:12.065132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:12.065096 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:12.794127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:12.794097 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:13.569781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:13.569746 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:14.244982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:14.244946 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:14.944609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:14.944575 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:15.609948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.609913 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs"] Oct 30 19:33:15.618328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.618302 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.625575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.625548 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-976b80-serving-cert\"" Oct 30 19:33:15.689320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.689286 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs"] Oct 30 19:33:15.794890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.781204 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.795358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.795323 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.810909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.795612 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5frxq\" (UniqueName: \"kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.811301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.811261 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.888938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.863377 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:15.914033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.913983 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.915660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.915562 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.920363 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.916413 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.924227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.924182 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5frxq\" (UniqueName: \"kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.924824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.924805 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.925215 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:33:15.925087 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-serving-cert: secret "maas-api-e2e-shared-b-w3-976b80-serving-cert" not found Oct 30 19:33:15.925655 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:33:15.925642 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls podName:93e751e3-0511-411a-a275-cb9359879170 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:16.425602602 +0000 UTC m=+966.383421775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls") pod "maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" (UID: "93e751e3-0511-411a-a275-cb9359879170") : secret "maas-api-e2e-shared-b-w3-976b80-serving-cert" not found Oct 30 19:33:15.926855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.926833 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:15.962252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:15.957915 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5frxq\" (UniqueName: \"kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:16.032775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:16.032722 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:16.438601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:16.438570 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:16.477451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:16.477418 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:16.514890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:16.514540 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:16.614887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:16.610221 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:17.082966 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:33:17.082930 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e751e3_0511_411a_a275_cb9359879170.slice/crio-4ffa18997ca6af5aa24f156ff8317920e80f43f1f135b125d486896b9a494a46 WatchSource:0}: Error finding container 4ffa18997ca6af5aa24f156ff8317920e80f43f1f135b125d486896b9a494a46: Status 404 returned error can't find the container with id 4ffa18997ca6af5aa24f156ff8317920e80f43f1f135b125d486896b9a494a46 Oct 30 19:33:17.083314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:17.083070 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs"] Oct 30 19:33:17.227936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:17.225768 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:17.260204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:17.260173 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" event={"ID":"93e751e3-0511-411a-a275-cb9359879170","Type":"ContainerStarted","Data":"4ffa18997ca6af5aa24f156ff8317920e80f43f1f135b125d486896b9a494a46"} Oct 30 19:33:18.132037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:18.132005 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:18.293942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:18.289255 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" event={"ID":"93e751e3-0511-411a-a275-cb9359879170","Type":"ContainerStarted","Data":"4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709"} Oct 30 19:33:18.303898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:18.296010 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:18.322678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:18.322596 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" podStartSLOduration=2.870050268 podStartE2EDuration="3.322580488s" totalImagesPullingTime="452.53022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:18.321235856 +0000 UTC m=+968.279055047" watchObservedRunningTime="2026-10-30 19:33:18.322580488 +0000 UTC m=+968.280399725" Oct 30 19:33:18.961154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:18.961116 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:19.636765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:19.636730 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:20.145475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:20.145449 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:33:20.391411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:20.390840 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:20.704727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:20.704692 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:20.892654 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:33:20.892618 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:33:21.127671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:21.127582 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:22.053361 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:22.053319 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:22.871565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:22.871432 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:23.229320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.229285 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56"] Oct 30 19:33:23.235708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.235686 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56"] Oct 30 19:33:23.235847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.235776 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.240586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.240563 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-20e66fa4\"/\"kube-root-ca.crt\"" Oct 30 19:33:23.242271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.242251 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-20e66fa4\"/\"default-dockercfg-kbct8\"" Oct 30 19:33:23.242393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.242293 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-20e66fa4\"/\"e2e-lifecycle-model-20e66fa4-kserve-self-signed-certs\"" Oct 30 19:33:23.242393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.242306 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-20e66fa4\"/\"openshift-service-ca.crt\"" Oct 30 19:33:23.285969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.285938 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:23.365345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.365311 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.365528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.365507 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.365650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.365634 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.365716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.365679 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.365777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.365741 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.365925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.365906 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8chh\" (UniqueName: \"kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.469094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.469048 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.469442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.469421 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.469639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.469621 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.469769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.469753 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.469984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.469964 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.470191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.470173 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8chh\" (UniqueName: \"kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.471241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.471215 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.472154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.472131 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.493039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.477627 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.493039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.477891 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.520820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.520782 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.521510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.521486 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8chh\" (UniqueName: \"kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh\") pod \"e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.552735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.552699 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:23.708481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:23.708443 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:24.044022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.043993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:24.148371 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:33:24.148342 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod655561b9_9d89_4e83_b0f6_43d74da56c1f.slice/crio-f111c3cb004e58e142ea7dac048ee86cf92e1f6e45c3e2730659318b786c4366 WatchSource:0}: Error finding container f111c3cb004e58e142ea7dac048ee86cf92e1f6e45c3e2730659318b786c4366: Status 404 returned error can't find the container with id f111c3cb004e58e142ea7dac048ee86cf92e1f6e45c3e2730659318b786c4366 Oct 30 19:33:24.148751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.148732 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56"] Oct 30 19:33:24.302538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.302512 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:33:24.349275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.349144 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" event={"ID":"655561b9-9d89-4e83-b0f6-43d74da56c1f","Type":"ContainerStarted","Data":"04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42"} Oct 30 19:33:24.349275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.349207 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" event={"ID":"655561b9-9d89-4e83-b0f6-43d74da56c1f","Type":"ContainerStarted","Data":"f111c3cb004e58e142ea7dac048ee86cf92e1f6e45c3e2730659318b786c4366"} Oct 30 19:33:24.403262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.403160 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podStartSLOduration=1.4031449409999999 podStartE2EDuration="1.403144941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:24.401335844 +0000 UTC m=+974.359155034" watchObservedRunningTime="2026-10-30 19:33:24.403144941 +0000 UTC m=+974.360964130" Oct 30 19:33:24.554204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.554173 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:24.636024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.635991 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:24.905905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:24.897759 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:25.354668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:25.354630 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:26.272890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:26.269926 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:27.173668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:27.173636 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:28.109620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:28.109592 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:33:28.445894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:28.442409 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:28.742221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:28.742124 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:28.920949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:28.914231 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:30.270057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.270023 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:30.478312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.478276 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:30.541986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.539583 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc"] Oct 30 19:33:30.546954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.546843 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:30.556344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.549669 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc"] Oct 30 19:33:30.612710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.612678 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:30.652277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.652234 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-59f017-dockercfg-78hxs\"" Oct 30 19:33:30.658219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.658185 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-59f017\"" Oct 30 19:33:30.729888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.701259 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:30.871294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:30.870973 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7696\" (UniqueName: \"kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696\") pod \"payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:31.062070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:31.061984 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:31.066015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:31.065962 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7696\" (UniqueName: \"kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696\") pod \"payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:31.091656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:31.070428 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:31.176419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:31.176240 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7696\" (UniqueName: \"kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696\") pod \"payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:31.183148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:31.183123 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:31.776186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:31.771027 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc"] Oct 30 19:33:32.016886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:32.016156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:32.814847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:32.814803 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" event={"ID":"30f68ca6-4f5d-4cff-8850-08260c8ede71","Type":"ContainerStarted","Data":"02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a"} Oct 30 19:33:32.815520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:32.815495 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" event={"ID":"30f68ca6-4f5d-4cff-8850-08260c8ede71","Type":"ContainerStarted","Data":"ee2683e05b95e0290da7c65c9e2d7539cf9b12d53dfca71420624940c842a40a"} Oct 30 19:33:32.817192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:32.817169 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:32.879124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:32.879058 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" podStartSLOduration=2.879039814 podStartE2EDuration="2.879039814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:32.87152352 +0000 UTC m=+982.829342712" watchObservedRunningTime="2026-10-30 19:33:32.879039814 +0000 UTC m=+982.836859004" Oct 30 19:33:33.267255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:33.267217 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:33.553809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:33.553706 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:33.566501 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:33.566475 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:33.837568 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:33.837493 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:33:34.155789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:34.155713 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:34.932670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:34.932634 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:35.648510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:35.648475 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:36.086140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:36.086106 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:36.532647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:36.532610 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:37.267699 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:37.267670 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:37.940366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:37.940332 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:38.087749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:38.087675 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:38.300616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:38.300443 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:39.090044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:39.090012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:39.879571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:39.879518 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:40.587339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.587262 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:40.909551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.909519 6514 generic.go:356] "Generic (PLEG): container finished" podID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerID="a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9" exitCode=0 Oct 30 19:33:40.910004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.909574 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" event={"ID":"78eb4588-c43b-4925-99c5-73c0183e10ba","Type":"ContainerDied","Data":"a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9"} Oct 30 19:33:40.910097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.910078 6514 scope.go:122] "RemoveContainer" containerID="a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9" Oct 30 19:33:40.912043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.912002 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:40.912148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.912086 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:40.912148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:40.912121 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:41.342514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:41.342478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:41.920218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:41.920172 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" event={"ID":"78eb4588-c43b-4925-99c5-73c0183e10ba","Type":"ContainerStarted","Data":"cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875"} Oct 30 19:33:41.964206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:41.964172 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd"] Oct 30 19:33:41.968246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:41.968215 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:41.980159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:41.977338 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-59f017-serving-cert\"" Oct 30 19:33:41.987106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:41.987069 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd"] Oct 30 19:33:42.031122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.031080 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.118943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.104299 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.118943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.104416 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przkq\" (UniqueName: \"kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.118943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.104464 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.118943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.104579 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.165441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.165358 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.210516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.208169 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.210516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.208249 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przkq\" (UniqueName: \"kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.210516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.208287 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.210516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.208389 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.210516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.209836 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.231073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.228769 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.240126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.240080 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przkq\" (UniqueName: \"kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.253251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.243768 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.281388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.281347 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.299507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.299211 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:42.535261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.535186 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:33:42.535467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.535442 6514 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:42.535526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.535502 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 19:33:42.812903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:42.809941 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd"] Oct 30 19:33:42.913894 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:33:42.913691 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8434eb4_8a1e_4d5d_befe_8e13091e911f.slice/crio-5394dbdfcb92e396c7e0cca9e0849b1389cdfccd6fbf98fd214e4adc9dbdc0a1 WatchSource:0}: Error finding container 5394dbdfcb92e396c7e0cca9e0849b1389cdfccd6fbf98fd214e4adc9dbdc0a1: Status 404 returned error can't find the container with id 5394dbdfcb92e396c7e0cca9e0849b1389cdfccd6fbf98fd214e4adc9dbdc0a1 Oct 30 19:33:43.003888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:43.001489 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" event={"ID":"e8434eb4-8a1e-4d5d-befe-8e13091e911f","Type":"ContainerStarted","Data":"5394dbdfcb92e396c7e0cca9e0849b1389cdfccd6fbf98fd214e4adc9dbdc0a1"} Oct 30 19:33:43.152235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:43.152144 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:43.536813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:43.536060 6514 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:43.536813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:43.536122 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 19:33:44.024961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.024923 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" event={"ID":"e8434eb4-8a1e-4d5d-befe-8e13091e911f","Type":"ContainerStarted","Data":"327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1"} Oct 30 19:33:44.025569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.025151 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:44.046812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.046755 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" podStartSLOduration=2.552599578 podStartE2EDuration="3.046742347s" totalImagesPullingTime="494.142769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:44.045263391 +0000 UTC m=+994.003082579" watchObservedRunningTime="2026-10-30 19:33:44.046742347 +0000 UTC m=+994.004561570" Oct 30 19:33:44.061160 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.061125 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:44.535164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.535131 6514 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:44.535348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.535192 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 19:33:44.755558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.755526 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:44.870936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:44.870080 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:33:45.535222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:45.535188 6514 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:45.535650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:45.535249 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 19:33:45.645552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:45.645518 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:46.436831 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:46.436799 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:46.541887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:46.540957 6514 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:46.541887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:46.541027 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 19:33:47.377098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:47.377065 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:47.541895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:47.540327 6514 patch_prober.go:29] interesting pod/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:47.541895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:47.540390 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 19:33:48.051994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:48.051902 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:48.062412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:48.062378 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:33:48.540152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:48.540126 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:33:48.748418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:48.748388 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:48.879669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:48.879586 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.047001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:49.046963 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.077325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:49.077301 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:33:49.554316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:49.554284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:50.063100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:50.063071 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:33:50.444339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:50.444311 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:50.884842 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:33:50.884752 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:33:51.140507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:51.140431 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:51.845148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:51.845114 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:52.619899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:52.617159 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:53.550156 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:53.550106 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.855550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:54.855513 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:55.958892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:55.958850 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:56.746931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:56.746898 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:57.197071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:57.197039 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:57.552271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:57.552164 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:58.247461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:58.247423 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:58.977738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:58.977705 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.121238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:59.115680 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.529579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:59.529535 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.647385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:33:59.647336 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:00.213940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:00.197800 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:00.539959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:00.539853 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:00.876112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:00.876039 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:01.714470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:01.714415 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:02.538576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:02.538546 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:03.557928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:03.557845 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:03.776554 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:03.776509 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:04.189546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:04.189501 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:04.329057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:04.329033 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:04.491936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:04.488489 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:05.506296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:05.506267 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:05.644305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:05.644264 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:06.696201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:06.696167 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:07.762832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:07.762803 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.542163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:08.542133 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:09.230064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:09.230022 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.040450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:10.040414 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.294186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:10.294109 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.736350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:10.736314 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:11.153151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:11.153079 6514 scope.go:122] "RemoveContainer" containerID="aeabf6c0fe08b88c5284247dfed932a3850d0e8f9d84d7a5bd4cf56aa2daf744" Oct 30 19:34:11.211573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:11.211543 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:11.388115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:11.388085 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:11.684377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:11.684340 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl"] Oct 30 19:34:11.773024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:11.772992 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:11.890088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:11.890059 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:12.090002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:12.089685 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:12.951919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:12.951879 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:13.753323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:13.753289 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:14.436579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:14.436544 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:15.330735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:15.330708 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:16.049520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:16.049488 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:16.260014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:16.259266 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:16.407083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:16.406991 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:16.684968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:16.684924 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" podUID="8a5bf917-3608-4595-b00e-5d18c6fc9e97" containerName="payload-processing" containerID="cri-o://0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10" gracePeriod=25 Oct 30 19:34:16.986975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:16.974402 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:17.161748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.161720 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:34:17.222413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.222372 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume\") pod \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " Oct 30 19:34:17.222708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.222693 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l\" (UniqueName: \"kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l\") pod \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\" (UID: \"8a5bf917-3608-4595-b00e-5d18c6fc9e97\") " Oct 30 19:34:17.222813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.222720 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume" pod "8a5bf917-3608-4595-b00e-5d18c6fc9e97" (UID: "8a5bf917-3608-4595-b00e-5d18c6fc9e97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:17.223227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.223210 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5bf917-3608-4595-b00e-5d18c6fc9e97-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:17.225301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.225277 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l" pod "8a5bf917-3608-4595-b00e-5d18c6fc9e97" (UID: "8a5bf917-3608-4595-b00e-5d18c6fc9e97"). InnerVolumeSpecName "kube-api-access-szn4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:17.255185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.255156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:17.363898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.359063 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szn4l\" (UniqueName: \"kubernetes.io/projected/8a5bf917-3608-4595-b00e-5d18c6fc9e97-kube-api-access-szn4l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:17.748716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.748671 6514 generic.go:356] "Generic (PLEG): container finished" podID="8a5bf917-3608-4595-b00e-5d18c6fc9e97" containerID="0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10" exitCode=0 Oct 30 19:34:17.748946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.748786 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" Oct 30 19:34:17.749043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.748846 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" event={"ID":"8a5bf917-3608-4595-b00e-5d18c6fc9e97","Type":"ContainerDied","Data":"0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10"} Oct 30 19:34:17.749124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.749105 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl" event={"ID":"8a5bf917-3608-4595-b00e-5d18c6fc9e97","Type":"ContainerDied","Data":"07a943537b574a2d7c431daefa9432d900654b14a9e8f365e393f917ced9bf9e"} Oct 30 19:34:17.749197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.749139 6514 scope.go:122] "RemoveContainer" containerID="0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10" Oct 30 19:34:17.760458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.760440 6514 scope.go:122] "RemoveContainer" containerID="0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10" Oct 30 19:34:17.760717 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:34:17.760700 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10\": container with ID starting with 0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10 not found: ID does not exist" containerID="0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10" Oct 30 19:34:17.760761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.760722 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10"} err="failed to get container status \"0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10\": rpc error: code = NotFound desc = could not find container \"0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10\": container with ID starting with 0ae5e72090507e3c95ca2d9b93271a05c77c3e42e485dfb5e9c938e1bf2d1f10 not found: ID does not exist" Oct 30 19:34:17.777740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.777718 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl"] Oct 30 19:34:17.789051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:17.789029 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2fd9a937-7b6c4cdc98-p4kpl"] Oct 30 19:34:18.034199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:18.034114 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:18.200521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:18.200483 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5bf917-3608-4595-b00e-5d18c6fc9e97" path="/var/lib/kubelet/pods/8a5bf917-3608-4595-b00e-5d18c6fc9e97/volumes" Oct 30 19:34:18.864408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:18.864378 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:19.561709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:19.561672 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:20.282057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:20.282015 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:20.890197 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:34:20.890163 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:34:20.930191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:20.930158 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:21.615612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:21.615579 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:22.366539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:22.366504 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:23.053417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:23.053387 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:23.647550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:23.647521 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:23.837262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:23.837225 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:24.117296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:24.108321 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:24.285564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:24.285531 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:24.682828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:24.682789 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:25.073989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:25.073902 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:25.759495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:25.759467 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:26.671889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:26.665012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:27.391107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:27.391065 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:27.504187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:27.504152 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:28.149568 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:28.149533 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:28.539077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:28.539047 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:28.658011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:28.657981 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:29.182654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:29.182623 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:29.320890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:29.319795 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:34:29.462325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:29.462236 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:30.587892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:30.583619 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:30.784888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:30.781345 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:30.913936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:30.912993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:31.040737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:31.040705 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:31.982044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:31.982006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.165196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:32.165155 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:33.774928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:33.749910 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.098995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.069943 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.098995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.085146 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:34:34.376013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.341653 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:34:34.528884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.524567 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:34:34.578901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.575521 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.604916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.599102 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:34:34.645054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.644951 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/health\": context deadline exceeded" Oct 30 19:34:34.746910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.746304 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:34:34.839310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:34.839268 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:34:35.296365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:35.296330 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:35.474477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:35.474437 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:35.612611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:35.612507 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:35.755608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:35.755574 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:36.771276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:36.771243 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:36.930900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:36.930417 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:37.545392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:37.545365 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:38.328454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:38.328418 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:39.177915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:39.177882 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:39.821643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:39.821604 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:39.953422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:39.953386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.125229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:40.125127 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.746035 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:40.746003 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:41.151925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:41.151833 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:41.279762 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:41.279685 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:41.976482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:41.976431 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.827603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:42.827570 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:43.559884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:43.559815 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.160318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:44.160284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.349938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:44.346167 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.963182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:44.963151 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.657582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.657522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.750886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.738930 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg"] Oct 30 19:34:45.752721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.752685 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8a5bf917-3608-4595-b00e-5d18c6fc9e97" containerName="payload-processing" Oct 30 19:34:45.781896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.777051 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5bf917-3608-4595-b00e-5d18c6fc9e97" containerName="payload-processing" Oct 30 19:34:45.782773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.782744 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5bf917-3608-4595-b00e-5d18c6fc9e97" containerName="payload-processing" Oct 30 19:34:45.828222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.828182 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg"] Oct 30 19:34:45.828420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.828376 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.833849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.833821 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-20e66fa4-openshift-default-dockercfg-7rz7c\"" Oct 30 19:34:45.917081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.916989 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917054 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkgd\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917266 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917299 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917320 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917377 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917461 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917558 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:45.917630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:45.917616 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019125 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019177 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019304 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019360 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019445 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019564 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019574 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019634 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019696 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019719 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019937 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkgd\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.019963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.019948 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.020218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.020193 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.020321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.020301 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.021571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.021549 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.022485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.022462 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.027505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.027487 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkgd\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.027609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.027592 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.161330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.161295 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:46.395020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.394975 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.551663 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:34:46.551550 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886c3ef7_0f8f_424f_8d2e_2ce2a41fd98a.slice/crio-94679fedf62363609a1f0de0c5c207f0664d3a443f00a81fec22714ef30a2792 WatchSource:0}: Error finding container 94679fedf62363609a1f0de0c5c207f0664d3a443f00a81fec22714ef30a2792: Status 404 returned error can't find the container with id 94679fedf62363609a1f0de0c5c207f0664d3a443f00a81fec22714ef30a2792 Oct 30 19:34:46.555796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.555676 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:34:46.556412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.556386 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:34:46.556781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.556677 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:34:46.558236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:46.558211 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg"] Oct 30 19:34:47.067857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:47.067807 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:47.233717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:47.233669 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" event={"ID":"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a","Type":"ContainerStarted","Data":"ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e"} Oct 30 19:34:47.233927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:47.233731 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" event={"ID":"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a","Type":"ContainerStarted","Data":"94679fedf62363609a1f0de0c5c207f0664d3a443f00a81fec22714ef30a2792"} Oct 30 19:34:47.260262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:47.260219 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podStartSLOduration=2.260206071 podStartE2EDuration="2.260206071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:47.258253333 +0000 UTC m=+1057.216072522" watchObservedRunningTime="2026-10-30 19:34:47.260206071 +0000 UTC m=+1057.218025260" Oct 30 19:34:47.876301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:47.876265 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.162155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:48.162083 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:48.163890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:48.163852 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:34:48.164011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:48.163917 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:34:48.800373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:48.800314 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.162438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:49.162353 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:34:49.162438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:49.162409 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:34:49.690373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:49.690341 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.241027 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:50.240995 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:34:50.241531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:50.241484 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:34:50.664239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:50.664123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.972013 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:34:50.971970 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:34:50.977138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:50.977108 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.162144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.162105 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:34:51.162310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.162158 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:34:51.388646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.388564 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9"] Oct 30 19:34:51.426139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.426105 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.428237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.428205 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9"] Oct 30 19:34:51.433950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.433926 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c7880b-serving-cert\"" Oct 30 19:34:51.448008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.447981 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66wj\" (UniqueName: \"kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.448176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.448158 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.448318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.448299 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.448429 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.448406 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.451525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.451477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.565754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.565194 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.566174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.566151 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.566698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.566667 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66wj\" (UniqueName: \"kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.567184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.567146 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.570100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.569757 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.571065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.570979 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.603889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.603783 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66wj\" (UniqueName: \"kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.605089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.605042 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.639944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.633918 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls\") pod \"maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:51.752513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:51.752234 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:52.418889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:52.403438 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:34:52.418889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:52.403501 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:34:52.675486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:52.675448 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9"] Oct 30 19:34:52.829735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:52.829700 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:53.162426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.162385 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:34:53.162630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.162456 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:34:53.474528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.474493 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" event={"ID":"ca76d12c-94cd-4b5f-8046-3ec07ad31de5","Type":"ContainerStarted","Data":"ef4d904afe25a6b1c649c3177ef59102bb65254e1c0b4c2acccae5910d65ea0a"} Oct 30 19:34:53.475005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.474570 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" event={"ID":"ca76d12c-94cd-4b5f-8046-3ec07ad31de5","Type":"ContainerStarted","Data":"f2634145451630286516fc23be79fd358ba8bc45d6b78f63399386d425acb94a"} Oct 30 19:34:53.475005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.474603 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:34:53.500344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.500293 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" podStartSLOduration=1.995520178 podStartE2EDuration="2.5002778s" totalImagesPullingTime="504.757622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:53.498052592 +0000 UTC m=+1063.455871781" watchObservedRunningTime="2026-10-30 19:34:53.5002778 +0000 UTC m=+1063.458096988" Oct 30 19:34:53.557387 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:53.557343 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:54.167241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:54.167208 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:54.167754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:54.167730 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:54.168441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:54.168422 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:34:54.242615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:54.242569 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:55.030819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:55.030781 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:55.856684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:55.856646 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:56.758392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:56.758363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:57.544615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:57.544583 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:58.241914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:58.241853 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.226001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:59.225967 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.489715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:34:59.489640 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:35:00.462087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:00.462053 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:01.291637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:01.291602 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:02.591888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:02.584011 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:02.775937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:02.775888 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:02.907475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:02.907382 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:03.873415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:03.873376 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.571059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:04.571025 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.346713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:05.346674 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:06.220067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:06.220030 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:07.289233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:07.289198 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.068583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:08.068536 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.740282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:08.740227 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:09.037696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.037612 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr"] Oct 30 19:35:09.041792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.041768 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5043d15a-5hfbr"] Oct 30 19:35:09.131716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.131678 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7162-119d36"] Oct 30 19:35:09.136700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.136678 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:09.141671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.141343 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvcnk\"" Oct 30 19:35:09.141962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.141941 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-119d36"] Oct 30 19:35:09.245118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.244901 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q62p\" (UniqueName: \"kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p\") pod \"test-curl-7162-119d36\" (UID: \"dda6a03c-cb15-495f-b8a0-af37e3e7953b\") " pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:09.346808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.346725 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q62p\" (UniqueName: \"kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p\") pod \"test-curl-7162-119d36\" (UID: \"dda6a03c-cb15-495f-b8a0-af37e3e7953b\") " pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:09.355089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.355060 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q62p\" (UniqueName: \"kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p\") pod \"test-curl-7162-119d36\" (UID: \"dda6a03c-cb15-495f-b8a0-af37e3e7953b\") " pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:09.447848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.447820 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:09.454503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.454478 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:09.874500 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:09.874240 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-119d36"] Oct 30 19:35:10.161633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.161560 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140956c2-0f99-4931-aec6-021b0b16f136" path="/var/lib/kubelet/pods/140956c2-0f99-4931-aec6-021b0b16f136/volumes" Oct 30 19:35:10.305399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.305356 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.354641 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.353392 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-119d36" event={"ID":"dda6a03c-cb15-495f-b8a0-af37e3e7953b","Type":"ContainerStarted","Data":"76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2"} Oct 30 19:35:10.354641 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.353470 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-119d36" event={"ID":"dda6a03c-cb15-495f-b8a0-af37e3e7953b","Type":"ContainerStarted","Data":"dba2cbac988c907e4528369d0e8e5558517375aa0b7c7af27c3d981328f0a095"} Oct 30 19:35:10.379288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.379224 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7162-119d36" podStartSLOduration=1.379203619 podStartE2EDuration="1.379203619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:10.376254289 +0000 UTC m=+1080.334073480" watchObservedRunningTime="2026-10-30 19:35:10.379203619 +0000 UTC m=+1080.337022811" Oct 30 19:35:10.428064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.427980 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.817439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:10.817394 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.063311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:11.063284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.104654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:11.104561 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7162-119d36"] Oct 30 19:35:11.176888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:11.171053 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7162-119d36"] Oct 30 19:35:11.330359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:11.330329 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.462923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:11.461856 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:12.058888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:12.050014 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:12.370569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:12.370484 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7162-119d36" podUID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" containerName="test-curl-7162-119d36" containerID="cri-o://76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2" gracePeriod=2 Oct 30 19:35:12.554981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:12.554948 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:13.715891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:13.714453 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:14.459581 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.459537 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:14.602537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.579481 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:14.748558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.748528 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:14.891469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.891436 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p\" (UniqueName: \"kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p\") pod \"dda6a03c-cb15-495f-b8a0-af37e3e7953b\" (UID: \"dda6a03c-cb15-495f-b8a0-af37e3e7953b\") " Oct 30 19:35:14.895143 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.895110 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p" pod "dda6a03c-cb15-495f-b8a0-af37e3e7953b" (UID: "dda6a03c-cb15-495f-b8a0-af37e3e7953b"). InnerVolumeSpecName "kube-api-access-5q62p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:14.930445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.930413 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:14.994840 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:14.994761 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q62p\" (UniqueName: \"kubernetes.io/projected/dda6a03c-cb15-495f-b8a0-af37e3e7953b-kube-api-access-5q62p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:15.061914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.061859 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:15.396752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.396663 6514 generic.go:356] "Generic (PLEG): container finished" podID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" containerID="76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2" exitCode=137 Oct 30 19:35:15.396752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.396706 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:15.397033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.396757 6514 scope.go:122] "RemoveContainer" containerID="76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2" Oct 30 19:35:15.408631 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.408613 6514 scope.go:122] "RemoveContainer" containerID="76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2" Oct 30 19:35:15.408971 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:35:15.408953 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2\": container with ID starting with 76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2 not found: ID does not exist" containerID="76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2" Oct 30 19:35:15.409048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.408979 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2"} err="failed to get container status \"76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2\": rpc error: code = NotFound desc = could not find container \"76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2\": container with ID starting with 76c6d1cbb58c32b953655b38bd54938d8dbdfda0a4522f289353018e1ccec9f2 not found: ID does not exist" Oct 30 19:35:15.410386 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:35:15.410359 6514 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7162-119d36\" is forbidden: User \"system:node:ip-10-0-138-233.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-233.us-east-2.compute.internal' and this object" podUID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" pod="opendatahub/test-curl-7162-119d36" Oct 30 19:35:15.543892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:15.543424 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.170444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:16.170409 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" path="/var/lib/kubelet/pods/dda6a03c-cb15-495f-b8a0-af37e3e7953b/volumes" Oct 30 19:35:16.195716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:16.195685 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.335656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:16.333671 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.527883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:16.527722 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:17.419646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:17.418912 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:18.209934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:18.209892 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.051151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:19.051100 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.849357 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:19.849328 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:20.546196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:20.546144 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:20.883711 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:35:20.883622 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:35:21.208712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:21.208679 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:21.973326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:21.973291 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.084586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:22.084555 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:23.090422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:23.090384 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:23.808924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:23.808844 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:24.551050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:24.551006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:25.328227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:25.328192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:26.082134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:26.082090 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:26.766065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:26.766031 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.804157 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:27.804099 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.924914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:27.924346 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.747565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:28.747538 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:29.544172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:29.544144 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.257119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:30.257073 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.956029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:30.955993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:31.107934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:31.107848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:31.307320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:31.307246 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:31.746482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:31.746446 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.609779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:32.609746 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:33.399766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:33.399736 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:34.185765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:34.185685 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:34.936636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:34.936600 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:35.577097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:35.577064 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:35.577473 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:35.577436 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:35:36.450270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:36.450236 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:37.558940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:37.558848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:38.347530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:38.347496 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:39.412569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:39.412535 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:40.026327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:40.025953 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4g9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:35:40.026327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:40.026027 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.094888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:40.093964 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.744913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:40.744877 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:40.749965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:40.749928 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.968813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:40.968781 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.271798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:41.271759 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.473877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:41.473805 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:42.338561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:42.338528 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:43.452350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:43.452198 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:44.431553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:44.431519 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:44.788123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:44.788032 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:44.900547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:44.900505 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:45.270343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:45.270290 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:45.408776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:45.408737 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.011789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.011754 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.105815 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.105776 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt"] Oct 30 19:35:46.106691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.106668 6514 cpu_manager.go:414] "containerMap: removing container" podUID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" containerName="test-curl-7162-119d36" Oct 30 19:35:46.106814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.106723 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" containerName="test-curl-7162-119d36" Oct 30 19:35:46.107043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.107024 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda6a03c-cb15-495f-b8a0-af37e3e7953b" containerName="test-curl-7162-119d36" Oct 30 19:35:46.113593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.113563 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.119194 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.119169 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt"] Oct 30 19:35:46.329958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.246685 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume\") pod \"payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.372889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.367059 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhr5\" (UniqueName: \"kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5\") pod \"payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.372889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.291259 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-73f96d-dockercfg-r4rtm\"" Oct 30 19:35:46.372889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.291351 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-73f96d\"" Oct 30 19:35:46.524922 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.508943 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume\") pod \"payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.561884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.553538 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume\") pod \"payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.561884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.553712 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhr5\" (UniqueName: \"kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5\") pod \"payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.636540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.636458 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhr5\" (UniqueName: \"kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5\") pod \"payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:46.734190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:46.734156 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:47.009706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:47.009671 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:47.043758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:47.043724 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt"] Oct 30 19:35:47.044385 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:35:47.044359 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba50c9b_2375_4cd6_9e30_f2f08b7a5428.slice/crio-146837ba75255ba6ba73a8c6168774c0af787686e37146a09a3d70d1fe269890 WatchSource:0}: Error finding container 146837ba75255ba6ba73a8c6168774c0af787686e37146a09a3d70d1fe269890: Status 404 returned error can't find the container with id 146837ba75255ba6ba73a8c6168774c0af787686e37146a09a3d70d1fe269890 Oct 30 19:35:47.140661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:47.140622 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" event={"ID":"cba50c9b-2375-4cd6-9e30-f2f08b7a5428","Type":"ContainerStarted","Data":"146837ba75255ba6ba73a8c6168774c0af787686e37146a09a3d70d1fe269890"} Oct 30 19:35:47.376979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:47.373487 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:48.150797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:48.150667 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" event={"ID":"cba50c9b-2375-4cd6-9e30-f2f08b7a5428","Type":"ContainerStarted","Data":"f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f"} Oct 30 19:35:48.150797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:48.150761 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:48.172149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:48.172094 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" podStartSLOduration=2.172079696 podStartE2EDuration="2.172079696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:48.168344478 +0000 UTC m=+1118.126163663" watchObservedRunningTime="2026-10-30 19:35:48.172079696 +0000 UTC m=+1118.129898945" Oct 30 19:35:48.620209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:48.620167 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.158543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:49.158493 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.481420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:49.481359 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:50.286732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:50.286702 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:50.886112 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:35:50.886073 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:35:50.962289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:50.962260 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:51.836359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:51.836287 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:52.645834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:52.645801 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:53.190663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:53.190627 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g"] Oct 30 19:35:53.191215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:53.190843 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" podUID="8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" containerName="maas-api" containerID="cri-o://21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439" gracePeriod=30 Oct 30 19:35:53.273590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:53.273556 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:53.561017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:53.545145 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:53.953603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:53.953555 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:54.067740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.067712 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:35:54.157120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.157091 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca\") pod \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " Oct 30 19:35:54.157314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.157185 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs\") pod \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " Oct 30 19:35:54.157479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.157449 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca" pod "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" (UID: "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:54.157555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.157506 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq\" (UniqueName: \"kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq\") pod \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " Oct 30 19:35:54.157651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.157637 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls\") pod \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\" (UID: \"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a\") " Oct 30 19:35:54.158488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.158463 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:54.158650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.158627 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:54.159898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.159838 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq" pod "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" (UID: "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a"). InnerVolumeSpecName "kube-api-access-h6lqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:54.159999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.159976 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs" pod "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" (UID: "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.160135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.160113 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls" pod "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" (UID: "8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:54.270940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.268304 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6lqq\" (UniqueName: \"kubernetes.io/projected/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-kube-api-access-h6lqq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:54.270940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.268344 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:54.270940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.268359 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:54.301679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.295031 6514 generic.go:356] "Generic (PLEG): container finished" podID="8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" containerID="21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439" exitCode=0 Oct 30 19:35:54.301679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.295335 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" event={"ID":"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a","Type":"ContainerDied","Data":"21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439"} Oct 30 19:35:54.301679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.295397 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" event={"ID":"8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a","Type":"ContainerDied","Data":"32313c2c946c7fcf4254913afcbb6629406c4b881e24dcdc84a2199b04da1316"} Oct 30 19:35:54.301679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.295429 6514 scope.go:122] "RemoveContainer" containerID="21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439" Oct 30 19:35:54.301679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.295694 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g" Oct 30 19:35:54.391618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.391448 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g"] Oct 30 19:35:54.400414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.400393 6514 scope.go:122] "RemoveContainer" containerID="21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439" Oct 30 19:35:54.400756 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:35:54.400733 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439\": container with ID starting with 21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439 not found: ID does not exist" containerID="21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439" Oct 30 19:35:54.400808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.400766 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439"} err="failed to get container status \"21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439\": rpc error: code = NotFound desc = could not find container \"21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439\": container with ID starting with 21a5bc234a918196db6b6f84750e001221eb405daf62eb995a4d82bee8dab439 not found: ID does not exist" Oct 30 19:35:54.402239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:54.402218 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-kpf9g"] Oct 30 19:35:55.208888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:55.205872 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:56.161765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:56.161727 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" path="/var/lib/kubelet/pods/8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a/volumes" Oct 30 19:35:56.313996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:56.311517 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:56.951450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:56.951417 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.070572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:57.070543 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.113123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:58.113090 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.289884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:58.286475 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.841626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:58.841588 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.163032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.162950 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:35:59.831460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.831426 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw"] Oct 30 19:35:59.832409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.832390 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" containerName="maas-api" Oct 30 19:35:59.832485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.832411 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" containerName="maas-api" Oct 30 19:35:59.832590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.832580 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4cbe2f-ecea-499d-b5ff-9fbc06ffc94a" containerName="maas-api" Oct 30 19:35:59.836845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.836821 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:35:59.859553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.859518 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw"] Oct 30 19:35:59.877697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.877665 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-73f96d-serving-cert\"" Oct 30 19:35:59.975888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.967115 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfs5\" (UniqueName: \"kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:35:59.975888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.967232 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:35:59.975888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.967446 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:35:59.975888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.967587 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:35:59.986903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:35:59.985647 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.076906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.075398 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.076906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.075574 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.076906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.075773 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfs5\" (UniqueName: \"kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.076906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.075856 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.076906 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:36:00.076630 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-serving-cert: secret "maas-api-e2e-api-a-73f96d-serving-cert" not found Oct 30 19:36:00.076906 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:36:00.076730 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls podName:ad8e8a4c-2fbc-4597-80bb-e72412918855 nodeName:}" failed. No retries permitted until 2026-10-30 19:36:00.576701162 +0000 UTC m=+1130.534520357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls") pod "maas-api-e2e-api-a-73f96d-7488969884-vgffw" (UID: "ad8e8a4c-2fbc-4597-80bb-e72412918855") : secret "maas-api-e2e-api-a-73f96d-serving-cert" not found Oct 30 19:36:00.080901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.077803 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.117935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.115306 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.139650 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.137707 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfs5\" (UniqueName: \"kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.164431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.164394 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.297076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.297044 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.584545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.584499 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.607890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.606938 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\") pod \"maas-api-e2e-api-a-73f96d-7488969884-vgffw\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:00.765136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:00.763503 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:01.123041 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:36:01.122999 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8e8a4c_2fbc_4597_80bb_e72412918855.slice/crio-d453e1e2bc6441bd4c2fb38d2ee111df8e4131d01c4c4cf50816922f86e416d2 WatchSource:0}: Error finding container d453e1e2bc6441bd4c2fb38d2ee111df8e4131d01c4c4cf50816922f86e416d2: Status 404 returned error can't find the container with id d453e1e2bc6441bd4c2fb38d2ee111df8e4131d01c4c4cf50816922f86e416d2 Oct 30 19:36:01.123154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:01.123050 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw"] Oct 30 19:36:01.124632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:01.124616 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:36:01.184946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:01.180301 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.451724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:01.451641 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" event={"ID":"ad8e8a4c-2fbc-4597-80bb-e72412918855","Type":"ContainerStarted","Data":"d453e1e2bc6441bd4c2fb38d2ee111df8e4131d01c4c4cf50816922f86e416d2"} Oct 30 19:36:01.669482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:01.669452 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.799090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:01.798984 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:02.081291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:02.081213 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:02.638780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:02.631218 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:02.890540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:02.890452 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:02.942886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:02.924798 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" event={"ID":"ad8e8a4c-2fbc-4597-80bb-e72412918855","Type":"ContainerStarted","Data":"b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36"} Oct 30 19:36:02.958946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:02.952482 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:02.980449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:02.980400 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" podStartSLOduration=3.522003796 podStartE2EDuration="3.980385323s" totalImagesPullingTime="458.381527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:02.96057656 +0000 UTC m=+1132.918395752" watchObservedRunningTime="2026-10-30 19:36:02.980385323 +0000 UTC m=+1132.938204512" Oct 30 19:36:03.858012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:03.857975 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:04.753269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:04.753238 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:05.535836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:05.535800 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.226629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:06.226587 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.954771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:06.954734 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:07.657617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:07.657582 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.586792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:08.586756 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.992041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:08.992010 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:36:09.360933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.360836 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv"] Oct 30 19:36:09.367192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.367165 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.374803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.374769 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-20e66fa4-serving-cert\"" Oct 30 19:36:09.385052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.385016 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv"] Oct 30 19:36:09.454308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.454275 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.454492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.454355 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.454546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.454484 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vvh\" (UniqueName: \"kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.454647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.454633 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.486931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.486878 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:09.561856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.561813 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.582560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.582519 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vvh\" (UniqueName: \"kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.585244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.585211 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.586297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.586264 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.587449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.587223 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.597968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.594029 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.605132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.605100 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.624236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.624072 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vvh\" (UniqueName: \"kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh\") pod \"maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:09.631400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.631363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:09.691164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:09.691134 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:10.088919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:10.088856 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv"] Oct 30 19:36:10.089798 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:36:10.089773 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72855b30_8e3c_4f98_9143_2e23c94f6129.slice/crio-6880527dfddcffc6199449719054365ff34d169c87ddfa8e74bb2ef2a69e10ec WatchSource:0}: Error finding container 6880527dfddcffc6199449719054365ff34d169c87ddfa8e74bb2ef2a69e10ec: Status 404 returned error can't find the container with id 6880527dfddcffc6199449719054365ff34d169c87ddfa8e74bb2ef2a69e10ec Oct 30 19:36:10.298969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:10.298920 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" event={"ID":"72855b30-8e3c-4f98-9143-2e23c94f6129","Type":"ContainerStarted","Data":"6880527dfddcffc6199449719054365ff34d169c87ddfa8e74bb2ef2a69e10ec"} Oct 30 19:36:10.639219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:10.639182 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.256251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:11.256215 6514 scope.go:122] "RemoveContainer" containerID="557783d7716400659ab0057bfe501c3b8f620a8aee4fd311928c78973e4164fd" Oct 30 19:36:11.265298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:11.265261 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" event={"ID":"72855b30-8e3c-4f98-9143-2e23c94f6129","Type":"ContainerStarted","Data":"5c5d7e323e93bab844c2c3a2f31f757c5be7d2df2db70dcc2a76da35724df290"} Oct 30 19:36:11.265443 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:11.265354 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:11.289641 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:11.289597 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" podStartSLOduration=1.786415095 podStartE2EDuration="2.289582179s" totalImagesPullingTime="503.167084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:11.285967736 +0000 UTC m=+1141.243786927" watchObservedRunningTime="2026-10-30 19:36:11.289582179 +0000 UTC m=+1141.247401368" Oct 30 19:36:11.374191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:11.374144 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.506943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:11.506255 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:12.298881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:12.298820 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:13.013889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:13.002052 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:13.639027 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:13.638993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:14.537628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:14.537597 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:15.464768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:15.464732 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:17.235495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:17.235459 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:17.282706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:17.282673 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:18.145257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:18.145225 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:19.029543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:19.029508 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:19.833276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:19.832403 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:20.410897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:20.407129 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:20.657666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:20.657629 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:20.886220 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:36:20.886176 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:36:21.009552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:21.009522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:21.453514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:21.453477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:21.685886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:21.675296 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:22.087894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:22.076442 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:22.565611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:22.565566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:23.258079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:23.258041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:24.025212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:24.025177 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:24.740111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:24.740078 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:24.992431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:24.992345 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:26.016269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:26.016237 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:26.222888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:26.222242 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:26.901852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:26.901817 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9"] Oct 30 19:36:26.902126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:26.902100 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" podUID="ca76d12c-94cd-4b5f-8046-3ec07ad31de5" containerName="maas-api" containerID="cri-o://ef4d904afe25a6b1c649c3177ef59102bb65254e1c0b4c2acccae5910d65ea0a" gracePeriod=30 Oct 30 19:36:26.975364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:26.975293 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:27.426733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:27.426694 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:27.598822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:27.596014 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.003248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.002627 6514 generic.go:356] "Generic (PLEG): container finished" podID="ca76d12c-94cd-4b5f-8046-3ec07ad31de5" containerID="ef4d904afe25a6b1c649c3177ef59102bb65254e1c0b4c2acccae5910d65ea0a" exitCode=0 Oct 30 19:36:28.003248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.003014 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" event={"ID":"ca76d12c-94cd-4b5f-8046-3ec07ad31de5","Type":"ContainerDied","Data":"ef4d904afe25a6b1c649c3177ef59102bb65254e1c0b4c2acccae5910d65ea0a"} Oct 30 19:36:28.010468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.010439 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:36:28.077523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.077260 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.142316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.142278 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj\" (UniqueName: \"kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj\") pod \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " Oct 30 19:36:28.142966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.142926 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs\") pod \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " Oct 30 19:36:28.143317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.143299 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca\") pod \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " Oct 30 19:36:28.155768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.155728 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca" pod "ca76d12c-94cd-4b5f-8046-3ec07ad31de5" (UID: "ca76d12c-94cd-4b5f-8046-3ec07ad31de5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:28.156494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.156472 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls\") pod \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\" (UID: \"ca76d12c-94cd-4b5f-8046-3ec07ad31de5\") " Oct 30 19:36:28.267830 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.229960 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls" pod "ca76d12c-94cd-4b5f-8046-3ec07ad31de5" (UID: "ca76d12c-94cd-4b5f-8046-3ec07ad31de5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:28.268139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.266903 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs" pod "ca76d12c-94cd-4b5f-8046-3ec07ad31de5" (UID: "ca76d12c-94cd-4b5f-8046-3ec07ad31de5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:28.268697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.268671 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:28.268851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.268834 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:28.269031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.269016 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:28.272654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.272621 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj" pod "ca76d12c-94cd-4b5f-8046-3ec07ad31de5" (UID: "ca76d12c-94cd-4b5f-8046-3ec07ad31de5"). InnerVolumeSpecName "kube-api-access-x66wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:28.374339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.365210 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.375600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.375200 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66wj\" (UniqueName: \"kubernetes.io/projected/ca76d12c-94cd-4b5f-8046-3ec07ad31de5-kube-api-access-x66wj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:28.758231 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:28.758192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:29.012624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.012548 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" Oct 30 19:36:29.012624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.012581 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9" event={"ID":"ca76d12c-94cd-4b5f-8046-3ec07ad31de5","Type":"ContainerDied","Data":"f2634145451630286516fc23be79fd358ba8bc45d6b78f63399386d425acb94a"} Oct 30 19:36:29.012835 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.012642 6514 scope.go:122] "RemoveContainer" containerID="ef4d904afe25a6b1c649c3177ef59102bb65254e1c0b4c2acccae5910d65ea0a" Oct 30 19:36:29.034697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.034669 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9"] Oct 30 19:36:29.038875 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.038846 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7880b-54ddb97777-7cvg9"] Oct 30 19:36:29.127678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.127642 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:29.657204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.657170 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7162-123a01"] Oct 30 19:36:29.658295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.658268 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ca76d12c-94cd-4b5f-8046-3ec07ad31de5" containerName="maas-api" Oct 30 19:36:29.658295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.658291 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca76d12c-94cd-4b5f-8046-3ec07ad31de5" containerName="maas-api" Oct 30 19:36:29.658527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.658513 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca76d12c-94cd-4b5f-8046-3ec07ad31de5" containerName="maas-api" Oct 30 19:36:29.663904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.663844 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:29.669499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.669465 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-123a01"] Oct 30 19:36:29.688354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.688324 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvcnk\"" Oct 30 19:36:29.812246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.812210 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nft9j\" (UniqueName: \"kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j\") pod \"test-curl-7162-123a01\" (UID: \"fc3b61ec-da0a-4185-87e3-9726254deb75\") " pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:29.914315 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.914228 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nft9j\" (UniqueName: \"kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j\") pod \"test-curl-7162-123a01\" (UID: \"fc3b61ec-da0a-4185-87e3-9726254deb75\") " pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:29.925623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.925581 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nft9j\" (UniqueName: \"kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j\") pod \"test-curl-7162-123a01\" (UID: \"fc3b61ec-da0a-4185-87e3-9726254deb75\") " pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:29.981539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:29.981501 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:30.027400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:30.027367 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:30.257995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:30.257957 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca76d12c-94cd-4b5f-8046-3ec07ad31de5" path="/var/lib/kubelet/pods/ca76d12c-94cd-4b5f-8046-3ec07ad31de5/volumes" Oct 30 19:36:30.285365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:30.285325 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7162-123a01"] Oct 30 19:36:30.287346 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:36:30.287320 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc3b61ec_da0a_4185_87e3_9726254deb75.slice/crio-51cfc1d422214bcc2a0f7100e0f7617134a7fd509da28e9bfeb15cabdae98f27 WatchSource:0}: Error finding container 51cfc1d422214bcc2a0f7100e0f7617134a7fd509da28e9bfeb15cabdae98f27: Status 404 returned error can't find the container with id 51cfc1d422214bcc2a0f7100e0f7617134a7fd509da28e9bfeb15cabdae98f27 Oct 30 19:36:30.838456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:30.838424 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:31.113939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:31.109996 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-123a01" event={"ID":"fc3b61ec-da0a-4185-87e3-9726254deb75","Type":"ContainerStarted","Data":"ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4"} Oct 30 19:36:31.113939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:31.110068 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7162-123a01" event={"ID":"fc3b61ec-da0a-4185-87e3-9726254deb75","Type":"ContainerStarted","Data":"51cfc1d422214bcc2a0f7100e0f7617134a7fd509da28e9bfeb15cabdae98f27"} Oct 30 19:36:31.147889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:31.145690 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7162-123a01" podStartSLOduration=2.145670554 podStartE2EDuration="2.145670554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:31.144410346 +0000 UTC m=+1161.102229536" watchObservedRunningTime="2026-10-30 19:36:31.145670554 +0000 UTC m=+1161.103489743" Oct 30 19:36:31.789231 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:31.789198 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:31.809005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:31.808974 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7162-123a01"] Oct 30 19:36:31.815159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:31.815133 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7162-123a01"] Oct 30 19:36:32.778187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:32.778161 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:32.927884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:32.927808 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:33.129713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:33.129614 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7162-123a01" podUID="fc3b61ec-da0a-4185-87e3-9726254deb75" containerName="test-curl-7162-123a01" containerID="cri-o://ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4" gracePeriod=2 Oct 30 19:36:34.031359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:34.031323 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.172754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:34.172716 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.871144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:34.871113 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:35.362782 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:35.362753 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:35.429437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:35.429411 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j\" (UniqueName: \"kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j\") pod \"fc3b61ec-da0a-4185-87e3-9726254deb75\" (UID: \"fc3b61ec-da0a-4185-87e3-9726254deb75\") " Oct 30 19:36:35.431584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:35.431555 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j" pod "fc3b61ec-da0a-4185-87e3-9726254deb75" (UID: "fc3b61ec-da0a-4185-87e3-9726254deb75"). InnerVolumeSpecName "kube-api-access-nft9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:35.531690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:35.531653 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nft9j\" (UniqueName: \"kubernetes.io/projected/fc3b61ec-da0a-4185-87e3-9726254deb75-kube-api-access-nft9j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:35.836826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:35.836708 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:36.162404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.162322 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3b61ec-da0a-4185-87e3-9726254deb75" path="/var/lib/kubelet/pods/fc3b61ec-da0a-4185-87e3-9726254deb75/volumes" Oct 30 19:36:36.189499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.189463 6514 generic.go:356] "Generic (PLEG): container finished" podID="fc3b61ec-da0a-4185-87e3-9726254deb75" containerID="ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4" exitCode=137 Oct 30 19:36:36.189653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.189505 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7162-123a01" Oct 30 19:36:36.189653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.189518 6514 scope.go:122] "RemoveContainer" containerID="ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4" Oct 30 19:36:36.200057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.200038 6514 scope.go:122] "RemoveContainer" containerID="ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4" Oct 30 19:36:36.200366 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:36:36.200344 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4\": container with ID starting with ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4 not found: ID does not exist" containerID="ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4" Oct 30 19:36:36.200427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.200375 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4"} err="failed to get container status \"ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4\": rpc error: code = NotFound desc = could not find container \"ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4\": container with ID starting with ee0566502e58e1be1726af6b34bae24940ec2fb1adb17aa1ae555982f35f98f4 not found: ID does not exist" Oct 30 19:36:36.418432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.418357 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:36.538408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.538371 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:36.708888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.708806 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:36.827899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:36.823457 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:37.894492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:37.894461 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:38.820122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:38.820071 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:39.368712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:39.368681 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:40.358886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:40.358845 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:41.154832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:41.154804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:41.940365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:41.940331 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.410509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.410409 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.524573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.524537 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.644886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.639352 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x"] Oct 30 19:36:42.644886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.639725 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" containerID="cri-o://e63a5a2047f335e4a9810191d411dcf35945d8b7142764ea03c5496f3f3d1f32" gracePeriod=30 Oct 30 19:36:42.822928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.808453 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.863971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.862542 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5"] Oct 30 19:36:42.883795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:42.864957 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" containerID="cri-o://dbefef822062e45f1eb7dfebefe50fd288cea05492990d7ef63556fc0bcb7756" gracePeriod=30 Oct 30 19:36:43.094586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.094496 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:43.244954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.242799 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j"] Oct 30 19:36:43.244954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.243123 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" containerID="cri-o://46de497fb81788d68f1914f9ec4a57a5f39b9e5a01d25e17a719cf1b8101c49e" gracePeriod=30 Oct 30 19:36:43.324693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.324659 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8"] Oct 30 19:36:43.326980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.325236 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" containerID="cri-o://4934838d90e3cf4bac17fe83d4d5ad0ad5c68d28fe5301e19477a2a1e1b8f6d0" gracePeriod=30 Oct 30 19:36:43.455421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.455387 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz"] Oct 30 19:36:43.455918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:43.455639 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" containerID="cri-o://a7c532649e420f8ff741c515543285bd8b1863e2739eb9922aeff25052d5e9d2" gracePeriod=30 Oct 30 19:36:44.008114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:44.008076 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:44.100892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:44.094008 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:36:44.100892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:44.094073 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:36:44.615754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:44.615715 6514 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:36:44.616242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:44.615774 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:36:44.828974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:44.828937 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:45.559553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:45.559522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:46.433335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:46.433303 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:47.377386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:47.377338 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:47.652625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:47.652532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:47.767207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:47.767164 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:48.532841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.532803 6514 generic.go:356] "Generic (PLEG): container finished" podID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerID="a7c532649e420f8ff741c515543285bd8b1863e2739eb9922aeff25052d5e9d2" exitCode=0 Oct 30 19:36:48.533015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.532896 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" event={"ID":"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663","Type":"ContainerDied","Data":"a7c532649e420f8ff741c515543285bd8b1863e2739eb9922aeff25052d5e9d2"} Oct 30 19:36:48.534737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.534672 6514 generic.go:356] "Generic (PLEG): container finished" podID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerID="dbefef822062e45f1eb7dfebefe50fd288cea05492990d7ef63556fc0bcb7756" exitCode=0 Oct 30 19:36:48.534822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.534746 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" event={"ID":"342f2142-4e28-4b9d-bcd0-03799519cbe7","Type":"ContainerDied","Data":"dbefef822062e45f1eb7dfebefe50fd288cea05492990d7ef63556fc0bcb7756"} Oct 30 19:36:48.534900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.534821 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" event={"ID":"342f2142-4e28-4b9d-bcd0-03799519cbe7","Type":"ContainerDied","Data":"1fd5b1b5460bbed06f701a3487bfb6c1cf707841f570c7feab74b7a3dbf8ee1a"} Oct 30 19:36:48.534900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.534850 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd5b1b5460bbed06f701a3487bfb6c1cf707841f570c7feab74b7a3dbf8ee1a" Oct 30 19:36:48.536321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.536300 6514 generic.go:356] "Generic (PLEG): container finished" podID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerID="e63a5a2047f335e4a9810191d411dcf35945d8b7142764ea03c5496f3f3d1f32" exitCode=0 Oct 30 19:36:48.536429 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.536378 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" event={"ID":"11dfe9fc-35dc-4437-8475-f31edf469aae","Type":"ContainerDied","Data":"e63a5a2047f335e4a9810191d411dcf35945d8b7142764ea03c5496f3f3d1f32"} Oct 30 19:36:48.536429 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.536422 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" event={"ID":"11dfe9fc-35dc-4437-8475-f31edf469aae","Type":"ContainerDied","Data":"510c944da25c046c8244765ab74b03a9b46431873a0402279f2195592dda4060"} Oct 30 19:36:48.536600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.536448 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="510c944da25c046c8244765ab74b03a9b46431873a0402279f2195592dda4060" Oct 30 19:36:48.537994 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.537974 6514 generic.go:356] "Generic (PLEG): container finished" podID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerID="46de497fb81788d68f1914f9ec4a57a5f39b9e5a01d25e17a719cf1b8101c49e" exitCode=0 Oct 30 19:36:48.538080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.538042 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" event={"ID":"40bebcec-3413-4a53-ad7c-5c73a996bcb3","Type":"ContainerDied","Data":"46de497fb81788d68f1914f9ec4a57a5f39b9e5a01d25e17a719cf1b8101c49e"} Oct 30 19:36:48.540116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.539934 6514 generic.go:356] "Generic (PLEG): container finished" podID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerID="4934838d90e3cf4bac17fe83d4d5ad0ad5c68d28fe5301e19477a2a1e1b8f6d0" exitCode=0 Oct 30 19:36:48.540116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.539997 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" event={"ID":"2e40281f-cfcb-458a-b4c9-3216c0489992","Type":"ContainerDied","Data":"4934838d90e3cf4bac17fe83d4d5ad0ad5c68d28fe5301e19477a2a1e1b8f6d0"} Oct 30 19:36:48.562545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.562239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:48.749481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.749104 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:36:48.751484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.751456 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:36:48.757446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.757417 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:36:48.831493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.831460 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.831757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.831732 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.831928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.831910 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.832026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.831968 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.832088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832016 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.832142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832112 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832178 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832268 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832306 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832347 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.832399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832385 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.832496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832455 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832500 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832556 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.832606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832598 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832649 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.832710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832698 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.832821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832726 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.832821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832805 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.832953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832840 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.832953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832928 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs\") pod \"342f2142-4e28-4b9d-bcd0-03799519cbe7\" (UID: \"342f2142-4e28-4b9d-bcd0-03799519cbe7\") " Oct 30 19:36:48.833058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.832991 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.833058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.833044 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.833154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.833096 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.833203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.833162 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.833256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.833237 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket\") pod \"11dfe9fc-35dc-4437-8475-f31edf469aae\" (UID: \"11dfe9fc-35dc-4437-8475-f31edf469aae\") " Oct 30 19:36:48.833314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.833297 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy\") pod \"2e40281f-cfcb-458a-b4c9-3216c0489992\" (UID: \"2e40281f-cfcb-458a-b4c9-3216c0489992\") " Oct 30 19:36:48.834463 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.834440 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.834804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.834784 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:48.835660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.835634 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.836472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.835852 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.838700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.838658 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.838976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.838782 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.839108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.838873 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.839229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.839207 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.839340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.839225 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.840317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.839454 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.840317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.839608 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:48.840317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.839746 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:48.840317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.839983 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.843170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.843041 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "kube-api-access-gtzdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:48.843298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.843274 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.844537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.844496 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.845041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.845016 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "kube-api-access-vwz2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:48.845822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.845796 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:48.851980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.851952 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:36:48.851980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.851977 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:48.852149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.851960 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "kube-api-access-82zkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:48.852149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.852083 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:36:48.852149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.852093 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.853282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.853250 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo" pod "342f2142-4e28-4b9d-bcd0-03799519cbe7" (UID: "342f2142-4e28-4b9d-bcd0-03799519cbe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:36:48.857439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.857417 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy" pod "11dfe9fc-35dc-4437-8475-f31edf469aae" (UID: "11dfe9fc-35dc-4437-8475-f31edf469aae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.857548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.857432 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:48.864479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.864453 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token" pod "2e40281f-cfcb-458a-b4c9-3216c0489992" (UID: "2e40281f-cfcb-458a-b4c9-3216c0489992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:48.936037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936005 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936034 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936043 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936051 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936061 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-kube-api-access-gtzdk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936069 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936078 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936088 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936096 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936104 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936113 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936121 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936130 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936138 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f2142-4e28-4b9d-bcd0-03799519cbe7-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936147 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936154 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936166 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f2142-4e28-4b9d-bcd0-03799519cbe7-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936173 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936182 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936190 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f2142-4e28-4b9d-bcd0-03799519cbe7-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936199 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwz2t\" (UniqueName: \"kubernetes.io/projected/11dfe9fc-35dc-4437-8475-f31edf469aae-kube-api-access-vwz2t\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936208 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936218 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zkr\" (UniqueName: \"kubernetes.io/projected/2e40281f-cfcb-458a-b4c9-3216c0489992-kube-api-access-82zkr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936226 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e40281f-cfcb-458a-b4c9-3216c0489992-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936234 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dfe9fc-35dc-4437-8475-f31edf469aae-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936244 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e40281f-cfcb-458a-b4c9-3216c0489992-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:48.936266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:48.936252 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dfe9fc-35dc-4437-8475-f31edf469aae-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.199061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.198981 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:36:49.252767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.252731 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.252949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.252829 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.252949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.252875 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.253071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253048 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253128 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253107 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253255 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253309 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253448 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253509 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253561 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs\") pod \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\" (UID: \"40bebcec-3413-4a53-ad7c-5c73a996bcb3\") " Oct 30 19:36:49.254485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.253775 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.255737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.255133 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.255737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.255160 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.255737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.255174 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.256386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.256350 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.256543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.256513 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.257487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.256884 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:49.258047 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.258022 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.259157 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.259124 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:36:49.259911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.259833 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j" pod "40bebcec-3413-4a53-ad7c-5c73a996bcb3" (UID: "40bebcec-3413-4a53-ad7c-5c73a996bcb3"). InnerVolumeSpecName "kube-api-access-kpp8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:49.271103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.271070 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.355703 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.355813 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.355905 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.355982 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356040 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356104 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356149 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356161 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356450 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356496 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy\") pod \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\" (UID: \"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663\") " Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.356649 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357355 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357378 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357392 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357407 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357422 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357437 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357453 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp8j\" (UniqueName: \"kubernetes.io/projected/40bebcec-3413-4a53-ad7c-5c73a996bcb3-kube-api-access-kpp8j\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357468 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bebcec-3413-4a53-ad7c-5c73a996bcb3-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357751 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.357808 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.358351 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.359079 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.360068 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "kube-api-access-wb47r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.360163 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:49.362911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.360657 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo" pod "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" (UID: "1d1a8b7d-519d-46f9-ac75-3cd27d1f1663"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462493 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462537 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb47r\" (UniqueName: \"kubernetes.io/projected/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-kube-api-access-wb47r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462554 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462569 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462585 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462599 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.468962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.462615 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:49.551485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.551457 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" Oct 30 19:36:49.551684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.551537 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j" event={"ID":"40bebcec-3413-4a53-ad7c-5c73a996bcb3","Type":"ContainerDied","Data":"259ca271c3ea152d893a3b1d06be4f77487d3387fe2dc6a1c9e9e6b995754aab"} Oct 30 19:36:49.552311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.552284 6514 scope.go:122] "RemoveContainer" containerID="46de497fb81788d68f1914f9ec4a57a5f39b9e5a01d25e17a719cf1b8101c49e" Oct 30 19:36:49.553902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.553849 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" Oct 30 19:36:49.554026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.553848 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8" event={"ID":"2e40281f-cfcb-458a-b4c9-3216c0489992","Type":"ContainerDied","Data":"e97d3b40dc408d8cf94a63776b5ccfd8d09843ad752d50f138e3d24eb964b6a3"} Oct 30 19:36:49.555825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.555806 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5" Oct 30 19:36:49.561424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.561404 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" Oct 30 19:36:49.568909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.562579 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz" event={"ID":"1d1a8b7d-519d-46f9-ac75-3cd27d1f1663","Type":"ContainerDied","Data":"8cc366c0b2ea260115f004a850f50dc47fe59c39f1201f8cce2c1c7ada62bc11"} Oct 30 19:36:49.568909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.562615 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x" Oct 30 19:36:49.569315 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.569011 6514 scope.go:122] "RemoveContainer" containerID="4934838d90e3cf4bac17fe83d4d5ad0ad5c68d28fe5301e19477a2a1e1b8f6d0" Oct 30 19:36:49.584211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.584184 6514 scope.go:122] "RemoveContainer" containerID="a7c532649e420f8ff741c515543285bd8b1863e2739eb9922aeff25052d5e9d2" Oct 30 19:36:49.594660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.594615 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8"] Oct 30 19:36:49.603410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.603387 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-fz2s8"] Oct 30 19:36:49.609739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.609715 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5"] Oct 30 19:36:49.613927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.613904 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-g8pb5"] Oct 30 19:36:49.622554 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.622529 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j"] Oct 30 19:36:49.626079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.626055 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-6lg8j"] Oct 30 19:36:49.645921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.645893 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x"] Oct 30 19:36:49.652563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.652539 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-kqc8x"] Oct 30 19:36:49.664457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.664437 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz"] Oct 30 19:36:49.669401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:49.669377 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-mkgcz"] Oct 30 19:36:50.184460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.184387 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:50.234592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.234555 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" path="/var/lib/kubelet/pods/11dfe9fc-35dc-4437-8475-f31edf469aae/volumes" Oct 30 19:36:50.244395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.235676 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" path="/var/lib/kubelet/pods/1d1a8b7d-519d-46f9-ac75-3cd27d1f1663/volumes" Oct 30 19:36:50.247095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.245482 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" path="/var/lib/kubelet/pods/2e40281f-cfcb-458a-b4c9-3216c0489992/volumes" Oct 30 19:36:50.247816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.247772 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" path="/var/lib/kubelet/pods/342f2142-4e28-4b9d-bcd0-03799519cbe7/volumes" Oct 30 19:36:50.248779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.248756 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" path="/var/lib/kubelet/pods/40bebcec-3413-4a53-ad7c-5c73a996bcb3/volumes" Oct 30 19:36:50.882607 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:36:50.882570 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:36:50.941897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:50.941846 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:51.648580 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:51.648541 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:52.490946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:52.489909 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:53.336531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:53.336499 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:54.009085 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:54.009054 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:54.759805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:54.759771 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:55.548507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:55.548474 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:56.347589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:56.347560 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:56.876335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:56.876298 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv"] Oct 30 19:36:56.876803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:56.876555 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" podUID="72855b30-8e3c-4f98-9143-2e23c94f6129" containerName="maas-api" containerID="cri-o://5c5d7e323e93bab844c2c3a2f31f757c5be7d2df2db70dcc2a76da35724df290" gracePeriod=30 Oct 30 19:36:56.944836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:56.944797 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.119943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.104067 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.259889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.243276 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.613191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.599541 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.710823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.710431 6514 generic.go:356] "Generic (PLEG): container finished" podID="72855b30-8e3c-4f98-9143-2e23c94f6129" containerID="5c5d7e323e93bab844c2c3a2f31f757c5be7d2df2db70dcc2a76da35724df290" exitCode=0 Oct 30 19:36:57.710823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.710683 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" event={"ID":"72855b30-8e3c-4f98-9143-2e23c94f6129","Type":"ContainerDied","Data":"5c5d7e323e93bab844c2c3a2f31f757c5be7d2df2db70dcc2a76da35724df290"} Oct 30 19:36:57.710823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.710747 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" event={"ID":"72855b30-8e3c-4f98-9143-2e23c94f6129","Type":"ContainerDied","Data":"6880527dfddcffc6199449719054365ff34d169c87ddfa8e74bb2ef2a69e10ec"} Oct 30 19:36:57.710823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.710777 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6880527dfddcffc6199449719054365ff34d169c87ddfa8e74bb2ef2a69e10ec" Oct 30 19:36:57.723386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.723281 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.727161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.727121 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:57.859071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.859023 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls\") pod \"72855b30-8e3c-4f98-9143-2e23c94f6129\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " Oct 30 19:36:57.859741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.859719 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh\" (UniqueName: \"kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh\") pod \"72855b30-8e3c-4f98-9143-2e23c94f6129\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " Oct 30 19:36:57.859979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.859958 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs\") pod \"72855b30-8e3c-4f98-9143-2e23c94f6129\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " Oct 30 19:36:57.860084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.860047 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca\") pod \"72855b30-8e3c-4f98-9143-2e23c94f6129\" (UID: \"72855b30-8e3c-4f98-9143-2e23c94f6129\") " Oct 30 19:36:57.860590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.860562 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca" pod "72855b30-8e3c-4f98-9143-2e23c94f6129" (UID: "72855b30-8e3c-4f98-9143-2e23c94f6129"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:57.861113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.861093 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72855b30-8e3c-4f98-9143-2e23c94f6129-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:57.863857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.863794 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls" pod "72855b30-8e3c-4f98-9143-2e23c94f6129" (UID: "72855b30-8e3c-4f98-9143-2e23c94f6129"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:57.889639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.889580 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs" pod "72855b30-8e3c-4f98-9143-2e23c94f6129" (UID: "72855b30-8e3c-4f98-9143-2e23c94f6129"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:57.893013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.892150 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh" pod "72855b30-8e3c-4f98-9143-2e23c94f6129" (UID: "72855b30-8e3c-4f98-9143-2e23c94f6129"). InnerVolumeSpecName "kube-api-access-h4vvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:57.962784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.962743 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:57.962784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.962786 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vvh\" (UniqueName: \"kubernetes.io/projected/72855b30-8e3c-4f98-9143-2e23c94f6129-kube-api-access-h4vvh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:57.963010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:57.962803 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72855b30-8e3c-4f98-9143-2e23c94f6129-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:58.728887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:58.724400 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" Oct 30 19:36:58.808891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:58.805131 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv"] Oct 30 19:36:58.812243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:58.811043 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv"] Oct 30 19:36:58.951828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:36:58.951793 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:00.135889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:00.131449 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:00.196984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:00.195852 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72855b30-8e3c-4f98-9143-2e23c94f6129" path="/var/lib/kubelet/pods/72855b30-8e3c-4f98-9143-2e23c94f6129/volumes" Oct 30 19:37:00.288556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:00.288512 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20e66fa4-694c9dfc54-v56nv" podUID="72855b30-8e3c-4f98-9143-2e23c94f6129" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.129:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:00.968886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:00.965078 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:01.736188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:01.736136 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:01.808970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:01.808933 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:37:02.644057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:02.644023 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56"] Oct 30 19:37:02.803713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:02.803671 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:02.941909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:02.941858 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:04.555091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:04.555050 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:05.635578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:05.635522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:06.538891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.538768 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:06.680004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.679956 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:06.902047 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.901942 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": context deadline exceeded" Oct 30 19:37:06.920298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.915953 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 19:37:06.954007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.920579 6514 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 19:37:06.954007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.921745 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:06.957243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:06.954979 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.038618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:07.038562 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.391921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:07.368022 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:37:07.391921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:07.382459 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:07.410033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:07.400763 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.852890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:07.836182 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:37:07.943642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:07.931834 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": context deadline exceeded" Oct 30 19:37:08.019982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.019947 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:37:08.020623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.020590 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.023055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.023024 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:37:08.120949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.096705 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.213889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.213232 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:37:08.213889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.213669 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx"] Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.412798 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414207 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414291 6514 cpu_manager.go:414] "containerMap: removing container" podUID="72855b30-8e3c-4f98-9143-2e23c94f6129" containerName="maas-api" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414315 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72855b30-8e3c-4f98-9143-2e23c94f6129" containerName="maas-api" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414356 6514 cpu_manager.go:414] "containerMap: removing container" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414365 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414377 6514 cpu_manager.go:414] "containerMap: removing container" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414385 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414460 6514 cpu_manager.go:414] "containerMap: removing container" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414468 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414507 6514 cpu_manager.go:414] "containerMap: removing container" podUID="fc3b61ec-da0a-4185-87e3-9726254deb75" containerName="test-curl-7162-123a01" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414515 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3b61ec-da0a-4185-87e3-9726254deb75" containerName="test-curl-7162-123a01" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414550 6514 cpu_manager.go:414] "containerMap: removing container" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414558 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414577 6514 cpu_manager.go:414] "containerMap: removing container" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.414585 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415340 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dfe9fc-35dc-4437-8475-f31edf469aae" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415364 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e40281f-cfcb-458a-b4c9-3216c0489992" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415401 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="72855b30-8e3c-4f98-9143-2e23c94f6129" containerName="maas-api" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415414 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1a8b7d-519d-46f9-ac75-3cd27d1f1663" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415431 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bebcec-3413-4a53-ad7c-5c73a996bcb3" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415455 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="342f2142-4e28-4b9d-bcd0-03799519cbe7" containerName="istio-proxy" Oct 30 19:37:08.419947 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.415475 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3b61ec-da0a-4185-87e3-9726254deb75" containerName="test-curl-7162-123a01" Oct 30 19:37:08.702979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.693013 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:08.759882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.749594 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:37:08.890097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.890065 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:08.963266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.897730 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:08.963612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.921753 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:37:08.963938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.942296 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 19:37:08.964417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.942538 6514 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-slbxc container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded" start-of-body= Oct 30 19:37:08.972511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:08.972466 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-slbxc" podUID="bfa4a876-df8d-45b5-aa59-37bf104db694" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded" Oct 30 19:37:09.028166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.028110 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.028617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.028582 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": context deadline exceeded" Oct 30 19:37:09.107790 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.107736 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.355608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.346640 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-82012b-openshift-default-dockercfg-5bwfq\"" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.534189 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlg75\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.534336 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562205 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx"] Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562320 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562362 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562407 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562441 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562480 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562513 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562566 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:37:09.573046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.562601 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:37:09.575995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.574226 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb" podUID="f37f7bbb-a255-4242-8332-98b97a2a0f68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:09.586746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.586708 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.589162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.589133 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.589432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.589407 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.589601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.589582 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.589735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.589717 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.590045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.590025 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.590221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.590203 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.761387 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.761530 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.761700 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.761773 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.761838 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.761907 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.762147 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.762242 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.762407 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlg75\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.763996 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.764258 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.765839 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.765817 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.777934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.772821 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.777934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.773452 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.777934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.775370 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:37:09.777934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.775418 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:37:09.788159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.788088 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.788950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.788915 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlg75\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.801795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.801759 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:09.802239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:09.802189 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token\") pod \"e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:10.048471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.048388 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:10.434310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.434280 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx"] Oct 30 19:37:10.444845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.444790 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:10.444963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.444922 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:10.445009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.444982 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:10.533907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.533860 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:37:10.535082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.535056 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:37:10.787360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.787257 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:10.944752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:10.943720 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:11.126808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.126714 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" event={"ID":"ab279feb-f6f1-4ea9-912a-8eef55566912","Type":"ContainerStarted","Data":"85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79"} Oct 30 19:37:11.127321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.126817 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" event={"ID":"ab279feb-f6f1-4ea9-912a-8eef55566912","Type":"ContainerStarted","Data":"1d84aee151b57da1fe383d229f41e468787bef5477852f1793cad35d8ca83280"} Oct 30 19:37:11.161065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.161011 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podStartSLOduration=4.160995735 podStartE2EDuration="4.160995735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:11.158276636 +0000 UTC m=+1201.116095826" watchObservedRunningTime="2026-10-30 19:37:11.160995735 +0000 UTC m=+1201.118814988" Oct 30 19:37:11.232569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.230424 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz"] Oct 30 19:37:11.243066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.242176 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.243432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.243390 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz"] Oct 30 19:37:11.248954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.248817 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-89a3f6-openshift-default-dockercfg-kjltj\"" Oct 30 19:37:11.297470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.297435 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:11.300404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.300377 6514 scope.go:122] "RemoveContainer" containerID="e63a5a2047f335e4a9810191d411dcf35945d8b7142764ea03c5496f3f3d1f32" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327105 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrqr\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327216 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327262 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327312 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327336 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327632 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327736 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327819 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.333889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.327905 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.471999 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472115 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472169 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472231 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472354 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrqr\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472411 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472445 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472487 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.478888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.472512 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.529885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.511965 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.529885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.520684 6514 scope.go:122] "RemoveContainer" containerID="2a679e611eacf2dc7fb1cdfcc6f2f6b1806007a60b70b9d205c0fdaba022215c" Oct 30 19:37:11.529885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.522746 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.529885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.523096 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.529885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.523381 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.529885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.528666 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.630747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.630711 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrqr\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.644600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.644512 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.647006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.646980 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.678095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.677834 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.789075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.789039 6514 scope.go:122] "RemoveContainer" containerID="dbefef822062e45f1eb7dfebefe50fd288cea05492990d7ef63556fc0bcb7756" Oct 30 19:37:11.839402 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.839365 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:11.866779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.866753 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:11.988886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:11.987063 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:12.050181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.050101 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:12.064126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.064096 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:12.140784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.140739 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:12.179567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.174693 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:12.179567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.176748 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:37:12.275330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.275295 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz"] Oct 30 19:37:12.290444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.290402 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:12.290701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.290675 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:12.290894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:12.290852 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:13.025612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.025580 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b"] Oct 30 19:37:13.025834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.025808 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" containerID="cri-o://9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d" gracePeriod=30 Oct 30 19:37:13.191895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.169954 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:13.244889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.240034 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" event={"ID":"8ba39fa3-0b84-4a9f-98b2-faad69a3720f","Type":"ContainerStarted","Data":"61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6"} Oct 30 19:37:13.244889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.240118 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" event={"ID":"8ba39fa3-0b84-4a9f-98b2-faad69a3720f","Type":"ContainerStarted","Data":"ff9dbd68488ae113d9ae216123652b7d328a1ebed6f5892722c4266139602972"} Oct 30 19:37:13.294571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.294466 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podStartSLOduration=2.294449089 podStartE2EDuration="2.294449089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:13.269364164 +0000 UTC m=+1203.227183355" watchObservedRunningTime="2026-10-30 19:37:13.294449089 +0000 UTC m=+1203.252268339" Oct 30 19:37:13.644345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.644265 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:13.834579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.834545 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:13.867009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.866987 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:13.872427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:13.872408 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:14.247603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:14.247573 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:14.248498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:14.248476 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:37:14.410235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:14.401145 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:15.084107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.084076 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:15.267716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.267685 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:15.344039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.337456 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj"] Oct 30 19:37:15.408223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.408184 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.448888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.445431 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-82012b\"" Oct 30 19:37:15.448888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.446180 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-82012b-dockercfg-sx2mn\"" Oct 30 19:37:15.448888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.446734 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj"] Oct 30 19:37:15.602967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.598991 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.602967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.599088 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtsr\" (UniqueName: \"kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr\") pod \"payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.708850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.705085 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.708850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.705161 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtsr\" (UniqueName: \"kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr\") pod \"payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.708850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.706316 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.731968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.723412 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtsr\" (UniqueName: \"kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr\") pod \"payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:15.890399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.890329 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:15.915520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:15.915487 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:16.026907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:16.026851 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:16.224894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:16.224002 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:16.250509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:16.249415 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj"] Oct 30 19:37:17.125704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.125658 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:17.271448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.271414 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:17.315793 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.313934 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4"] Oct 30 19:37:17.321458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.321428 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.322684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.322660 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4"] Oct 30 19:37:17.325372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.325342 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" event={"ID":"9b528a4e-8af9-49d2-9ebf-da6359f9b06b","Type":"ContainerStarted","Data":"cb230dd6badbdba1e42d37339680e99f6da111e51d7052e7310c4a02c7909456"} Oct 30 19:37:17.325481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.325389 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" event={"ID":"9b528a4e-8af9-49d2-9ebf-da6359f9b06b","Type":"ContainerStarted","Data":"3a7608aac584c64b9e2c405b849c6edabe726260423dbce0a55b6866b08ce6e0"} Oct 30 19:37:17.325556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.325535 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:17.410941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.402280 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-89a3f6\"" Oct 30 19:37:17.410941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.402446 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-89a3f6-dockercfg-lngqf\"" Oct 30 19:37:17.440037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.439071 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwg2r\" (UniqueName: \"kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r\") pod \"payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.440037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.439452 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.522887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.520937 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" podStartSLOduration=2.520914438 podStartE2EDuration="2.520914438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:17.419171638 +0000 UTC m=+1207.376990827" watchObservedRunningTime="2026-10-30 19:37:17.520914438 +0000 UTC m=+1207.478733629" Oct 30 19:37:17.573892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.541705 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.573892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.541952 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg2r\" (UniqueName: \"kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r\") pod \"payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.573892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.543515 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.673828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.672026 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" containerID="cri-o://04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42" gracePeriod=45 Oct 30 19:37:17.707848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.703786 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwg2r\" (UniqueName: \"kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r\") pod \"payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.802853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.802598 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:17.834301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:17.834237 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": dial tcp 10.134.0.121:8000: connect: connection refused" Oct 30 19:37:18.200458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.200435 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:37:18.272923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.265640 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4"] Oct 30 19:37:18.272923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.266144 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.298318 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir\") pod \"655561b9-9d89-4e83-b0f6-43d74da56c1f\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.298393 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh\" (UniqueName: \"kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh\") pod \"655561b9-9d89-4e83-b0f6-43d74da56c1f\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.298838 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home\") pod \"655561b9-9d89-4e83-b0f6-43d74da56c1f\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.299016 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm\") pod \"655561b9-9d89-4e83-b0f6-43d74da56c1f\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.299083 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs\") pod \"655561b9-9d89-4e83-b0f6-43d74da56c1f\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.299113 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache\") pod \"655561b9-9d89-4e83-b0f6-43d74da56c1f\" (UID: \"655561b9-9d89-4e83-b0f6-43d74da56c1f\") " Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.299159 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir" pod "655561b9-9d89-4e83-b0f6-43d74da56c1f" (UID: "655561b9-9d89-4e83-b0f6-43d74da56c1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.299917 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.300143 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache" pod "655561b9-9d89-4e83-b0f6-43d74da56c1f" (UID: "655561b9-9d89-4e83-b0f6-43d74da56c1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.303620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.300337 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home" pod "655561b9-9d89-4e83-b0f6-43d74da56c1f" (UID: "655561b9-9d89-4e83-b0f6-43d74da56c1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.307723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.307688 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs" pod "655561b9-9d89-4e83-b0f6-43d74da56c1f" (UID: "655561b9-9d89-4e83-b0f6-43d74da56c1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:18.307959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.307910 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh" pod "655561b9-9d89-4e83-b0f6-43d74da56c1f" (UID: "655561b9-9d89-4e83-b0f6-43d74da56c1f"). InnerVolumeSpecName "kube-api-access-t8chh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.309805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.309676 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm" pod "655561b9-9d89-4e83-b0f6-43d74da56c1f" (UID: "655561b9-9d89-4e83-b0f6-43d74da56c1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.333186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.333149 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:37:18.362075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.362039 6514 generic.go:356] "Generic (PLEG): container finished" podID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerID="9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d" exitCode=0 Oct 30 19:37:18.362311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.362291 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" Oct 30 19:37:18.362706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.362681 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" event={"ID":"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf","Type":"ContainerDied","Data":"9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d"} Oct 30 19:37:18.362756 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.362741 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b" event={"ID":"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf","Type":"ContainerDied","Data":"43b08dfdd1a484715547876cdf82b1ba917d2ec97dd0b59c641a22b20b871907"} Oct 30 19:37:18.362792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.362772 6514 scope.go:122] "RemoveContainer" containerID="9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d" Oct 30 19:37:18.367458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.367441 6514 generic.go:356] "Generic (PLEG): container finished" podID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerID="04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42" exitCode=0 Oct 30 19:37:18.367734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.367488 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" Oct 30 19:37:18.369296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.367504 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" event={"ID":"655561b9-9d89-4e83-b0f6-43d74da56c1f","Type":"ContainerDied","Data":"04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42"} Oct 30 19:37:18.369378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.369314 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56" event={"ID":"655561b9-9d89-4e83-b0f6-43d74da56c1f","Type":"ContainerDied","Data":"f111c3cb004e58e142ea7dac048ee86cf92e1f6e45c3e2730659318b786c4366"} Oct 30 19:37:18.371299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.371275 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" event={"ID":"5f21635e-9c48-466f-9233-3adc4b207c06","Type":"ContainerStarted","Data":"9cfd52979cadb852d0596eb7bde4ac8769c322928923de3082fe46d49c00baea"} Oct 30 19:37:18.392932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.392737 6514 scope.go:122] "RemoveContainer" containerID="9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d" Oct 30 19:37:18.393929 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:18.393648 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d\": container with ID starting with 9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d not found: ID does not exist" containerID="9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d" Oct 30 19:37:18.393929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.393685 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d"} err="failed to get container status \"9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d\": rpc error: code = NotFound desc = could not find container \"9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d\": container with ID starting with 9522e0aa75f1c99c0c91a6ce7403357255a51fe90dc28d5d3a1ee8e844e7588d not found: ID does not exist" Oct 30 19:37:18.393929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.393708 6514 scope.go:122] "RemoveContainer" containerID="04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42" Oct 30 19:37:18.400531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400502 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.400642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400572 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.400703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400670 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.400764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400727 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.400824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400763 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.401002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400887 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.401002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.400922 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.401120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.401030 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.401180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.401157 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo\") pod \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\" (UID: \"3e1f2674-c231-4bf7-8d3c-d11051d0b9bf\") " Oct 30 19:37:18.402022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.401997 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.402111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.402026 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.402111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.402044 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/655561b9-9d89-4e83-b0f6-43d74da56c1f-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.402111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.402057 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/655561b9-9d89-4e83-b0f6-43d74da56c1f-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.402111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.402072 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8chh\" (UniqueName: \"kubernetes.io/projected/655561b9-9d89-4e83-b0f6-43d74da56c1f-kube-api-access-t8chh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.404626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.404460 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "kube-api-access-d8msn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.404902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.404844 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.405009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.404961 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.405187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.405168 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.405297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.405273 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.405366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.405352 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:18.405618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.405598 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.406407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.406383 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.407748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.406972 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo" pod "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" (UID: "3e1f2674-c231-4bf7-8d3c-d11051d0b9bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:18.415592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.415520 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56"] Oct 30 19:37:18.415796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.415732 6514 scope.go:122] "RemoveContainer" containerID="04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42" Oct 30 19:37:18.416171 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:18.416151 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42\": container with ID starting with 04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42 not found: ID does not exist" containerID="04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42" Oct 30 19:37:18.416284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.416176 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42"} err="failed to get container status \"04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42\": rpc error: code = NotFound desc = could not find container \"04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42\": container with ID starting with 04920e0330eed50cbe54f4f1a16f510f233889a99d521534e4d4d1c108dbaa42 not found: ID does not exist" Oct 30 19:37:18.422257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.422230 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-20e66fa4/e2e-lifecycle-model-20e66fa4-kserve-7db6f457c-ggk56"] Oct 30 19:37:18.507957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.507832 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.511958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.511924 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.512183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.512163 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.512286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.512274 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.525164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.525117 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.525499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.525470 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.525634 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.525620 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.525750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.525737 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.525842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.525830 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8msn\" (UniqueName: \"kubernetes.io/projected/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf-kube-api-access-d8msn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.689352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.689318 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b"] Oct 30 19:37:18.693064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:18.693040 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-fmf6b"] Oct 30 19:37:19.273513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.273467 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:19.387206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.387171 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" event={"ID":"5f21635e-9c48-466f-9233-3adc4b207c06","Type":"ContainerStarted","Data":"209f8e40484aa8ebfd6a8d24e3c32a5a282fce038fc34dd7cae55a36bdea4121"} Oct 30 19:37:19.387369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.387255 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:19.389169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.389143 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt"] Oct 30 19:37:19.409422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.409367 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" podStartSLOduration=2.409350892 podStartE2EDuration="2.409350892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:19.409114624 +0000 UTC m=+1209.366933814" watchObservedRunningTime="2026-10-30 19:37:19.409350892 +0000 UTC m=+1209.367170083" Oct 30 19:37:19.410670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.409790 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:19.602616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:19.602526 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:20.215755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:20.215718 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" path="/var/lib/kubelet/pods/3e1f2674-c231-4bf7-8d3c-d11051d0b9bf/volumes" Oct 30 19:37:20.216390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:20.216345 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" path="/var/lib/kubelet/pods/655561b9-9d89-4e83-b0f6-43d74da56c1f/volumes" Oct 30 19:37:20.613636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:20.613494 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:20.884320 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:20.884241 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:37:21.526301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:21.526262 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:22.410133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:22.410090 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:23.322747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:23.322718 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.028495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.028467 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.318530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.318415 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb"] Oct 30 19:37:24.390387 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.390355 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" podUID="cba50c9b-2375-4cd6-9e30-f2f08b7a5428" containerName="payload-processing" containerID="cri-o://f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f" gracePeriod=25 Oct 30 19:37:24.489201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.489167 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.691804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.691776 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:37:24.803756 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.803724 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5\" (UniqueName: \"kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5\") pod \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " Oct 30 19:37:24.803945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.803782 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume\") pod \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\" (UID: \"cba50c9b-2375-4cd6-9e30-f2f08b7a5428\") " Oct 30 19:37:24.804219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.804187 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume" pod "cba50c9b-2375-4cd6-9e30-f2f08b7a5428" (UID: "cba50c9b-2375-4cd6-9e30-f2f08b7a5428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:24.804584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.804566 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:24.805956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.805928 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5" pod "cba50c9b-2375-4cd6-9e30-f2f08b7a5428" (UID: "cba50c9b-2375-4cd6-9e30-f2f08b7a5428"). InnerVolumeSpecName "kube-api-access-xbhr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:24.906426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:24.906342 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhr5\" (UniqueName: \"kubernetes.io/projected/cba50c9b-2375-4cd6-9e30-f2f08b7a5428-kube-api-access-xbhr5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:25.101510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.101480 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:25.498739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.498707 6514 generic.go:356] "Generic (PLEG): container finished" podID="cba50c9b-2375-4cd6-9e30-f2f08b7a5428" containerID="f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f" exitCode=0 Oct 30 19:37:25.498965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.498755 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" Oct 30 19:37:25.498965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.498805 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" event={"ID":"cba50c9b-2375-4cd6-9e30-f2f08b7a5428","Type":"ContainerDied","Data":"f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f"} Oct 30 19:37:25.498965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.498856 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt" event={"ID":"cba50c9b-2375-4cd6-9e30-f2f08b7a5428","Type":"ContainerDied","Data":"146837ba75255ba6ba73a8c6168774c0af787686e37146a09a3d70d1fe269890"} Oct 30 19:37:25.498965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.498904 6514 scope.go:122] "RemoveContainer" containerID="f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f" Oct 30 19:37:25.510453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.510435 6514 scope.go:122] "RemoveContainer" containerID="f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f" Oct 30 19:37:25.510720 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:25.510700 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f\": container with ID starting with f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f not found: ID does not exist" containerID="f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f" Oct 30 19:37:25.510787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.510725 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f"} err="failed to get container status \"f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f\": rpc error: code = NotFound desc = could not find container \"f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f\": container with ID starting with f280bd2af6d29aba5bab3bb2da122d8ccdc2d6a45418c37d4b0d012a2d2bf28f not found: ID does not exist" Oct 30 19:37:25.540071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.540044 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt"] Oct 30 19:37:25.543804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.543780 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73f96d-767bd9568-4pjpt"] Oct 30 19:37:25.857395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:25.857316 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.069900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:26.068478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.221937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:26.221903 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba50c9b-2375-4cd6-9e30-f2f08b7a5428" path="/var/lib/kubelet/pods/cba50c9b-2375-4cd6-9e30-f2f08b7a5428/volumes" Oct 30 19:37:26.422180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:26.422148 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.576780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:26.576668 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.942729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:26.942699 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:27.588578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:27.586832 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:27.919582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:27.919497 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45"] Oct 30 19:37:27.919751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:27.919722 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" containerID="cri-o://87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f" gracePeriod=30 Oct 30 19:37:28.059118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:28.059067 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl"] Oct 30 19:37:28.059326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:28.059283 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" containerID="cri-o://c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246" gracePeriod=30 Oct 30 19:37:28.375406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:28.375377 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:37:28.495664 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:28.495634 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm"] Oct 30 19:37:28.495859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:28.495836 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" containerID="cri-o://9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0" gracePeriod=30 Oct 30 19:37:28.842161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:28.842124 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.479540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.479496 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65"] Oct 30 19:37:29.480795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.480770 6514 cpu_manager.go:414] "containerMap: removing container" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" Oct 30 19:37:29.480919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.480798 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" Oct 30 19:37:29.480919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.480878 6514 cpu_manager.go:414] "containerMap: removing container" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" Oct 30 19:37:29.480919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.480887 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" Oct 30 19:37:29.481029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.480924 6514 cpu_manager.go:414] "containerMap: removing container" podUID="cba50c9b-2375-4cd6-9e30-f2f08b7a5428" containerName="payload-processing" Oct 30 19:37:29.481029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.480934 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba50c9b-2375-4cd6-9e30-f2f08b7a5428" containerName="payload-processing" Oct 30 19:37:29.481129 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.481115 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="655561b9-9d89-4e83-b0f6-43d74da56c1f" containerName="main" Oct 30 19:37:29.481164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.481142 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1f2674-c231-4bf7-8d3c-d11051d0b9bf" containerName="istio-proxy" Oct 30 19:37:29.481164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.481157 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba50c9b-2375-4cd6-9e30-f2f08b7a5428" containerName="payload-processing" Oct 30 19:37:29.485748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.485724 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.494742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.494701 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65"] Oct 30 19:37:29.497022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.496970 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-82012b-serving-cert\"" Oct 30 19:37:29.585583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.585540 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.607339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.607313 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j"] Oct 30 19:37:29.629088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.629051 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.629727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.629697 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs"] Oct 30 19:37:29.631807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.631778 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znqj\" (UniqueName: \"kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.632089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.632061 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.632254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.632160 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.715560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.715525 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.734252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.734173 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.734543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.734520 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.734683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.734663 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znqj\" (UniqueName: \"kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.734760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.734731 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.735287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.735258 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.737388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.737364 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.737546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.737524 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.744472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.744445 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znqj\" (UniqueName: \"kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj\") pod \"maas-api-e2e-api-a-82012b-64878c9fcb-gvv65\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.816796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.816759 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:29.827745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:29.827714 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.268008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.266402 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65"] Oct 30 19:37:30.404904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.403884 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.447617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.443048 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:37:30.563466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.563372 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.593427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.593391 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" event={"ID":"393374a1-9e7d-4867-a325-7dfd58126370","Type":"ContainerStarted","Data":"d563f2a4196ce7a5c1ff5689dbebefe266a10f20021d9c40d53062105ef378f5"} Oct 30 19:37:30.799676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.798949 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.940932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.940901 6514 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:37:30.941145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:30.940975 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:37:31.427259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:31.427226 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:31.602247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:31.602210 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" event={"ID":"393374a1-9e7d-4867-a325-7dfd58126370","Type":"ContainerStarted","Data":"79bd292c98af0c9836bb415bb4d415de97597a28b410ebcade083415e743c080"} Oct 30 19:37:31.602421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:31.602282 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:31.625399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:31.625352 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" podStartSLOduration=1.994388759 podStartE2EDuration="2.625338627s" totalImagesPullingTime="630.949868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:31.622575249 +0000 UTC m=+1221.580394441" watchObservedRunningTime="2026-10-30 19:37:31.625338627 +0000 UTC m=+1221.583157815" Oct 30 19:37:32.097733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.097697 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g"] Oct 30 19:37:32.103554 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.103522 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.109260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.109232 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-89a3f6-serving-cert\"" Oct 30 19:37:32.116799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.116755 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g"] Oct 30 19:37:32.178060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.178021 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:32.249060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.226524 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.249060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.226679 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.249060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.226782 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.249060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.226813 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djrvr\" (UniqueName: \"kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.301888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.301763 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:32.343884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.343011 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.343884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.343207 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.343884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.343391 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.343884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.343433 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djrvr\" (UniqueName: \"kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.344300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.344273 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.344498 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:32.344474 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-serving-cert: secret "maas-api-e2e-inf-a-89a3f6-serving-cert" not found Oct 30 19:37:32.344740 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:32.344726 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls podName:6119e345-2afa-428d-9cc6-f1d2f9140389 nodeName:}" failed. No retries permitted until 2026-10-30 19:37:32.844703205 +0000 UTC m=+1222.802522385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls") pod "maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" (UID: "6119e345-2afa-428d-9cc6-f1d2f9140389") : secret "maas-api-e2e-inf-a-89a3f6-serving-cert" not found Oct 30 19:37:32.359044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.358600 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djrvr\" (UniqueName: \"kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.359212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.359090 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.502295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.502253 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:32.868889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.868412 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:32.880886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:32.879606 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\") pod \"maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:33.032476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.032447 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:33.271408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.271378 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:37:33.392827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.392793 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394016 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g"] Oct 30 19:37:33.394181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394159 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394258 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394438 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394494 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394614 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394689 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394665 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394713 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.394814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.394800 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data\") pod \"db9eeec3-9f96-483e-ae46-54ff7776dd06\" (UID: \"db9eeec3-9f96-483e-ae46-54ff7776dd06\") " Oct 30 19:37:33.398439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.398357 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.402333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.401028 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.402333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.401059 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "kube-api-access-t75g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.402333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.401463 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.402333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.402186 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.402333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.402213 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.404222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.404197 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.407082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407060 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9eeec3-9f96-483e-ae46-54ff7776dd06-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.407179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407089 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.407179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407104 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.407179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407120 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.407179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407140 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.407179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407156 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75g6\" (UniqueName: \"kubernetes.io/projected/db9eeec3-9f96-483e-ae46-54ff7776dd06-kube-api-access-t75g6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.407179 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.407170 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.418061 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.418026 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.420727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.420699 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy" pod "db9eeec3-9f96-483e-ae46-54ff7776dd06" (UID: "db9eeec3-9f96-483e-ae46-54ff7776dd06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.442847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.442821 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:33.451905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.451850 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:37:33.509376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.509308 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.509376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.509346 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9eeec3-9f96-483e-ae46-54ff7776dd06-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.649106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.649008 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.653944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.653903 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.698677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.652409 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.699171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.699137 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.700713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.700686 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.703128 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.703213 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.703243 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.703311 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.703417 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy\") pod \"85766866-9614-45d4-a5b3-fba1fcae567d\" (UID: \"85766866-9614-45d4-a5b3-fba1fcae567d\") " Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.704251 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.704476 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85766866-9614-45d4-a5b3-fba1fcae567d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.708002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.704503 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.708722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.708318 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.708722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.708422 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.711988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.711893 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.723257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.723217 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.723485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.723238 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "kube-api-access-d786x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.723604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.723303 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.724148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.724123 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy" pod "85766866-9614-45d4-a5b3-fba1fcae567d" (UID: "85766866-9614-45d4-a5b3-fba1fcae567d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.724890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.724849 6514 generic.go:356] "Generic (PLEG): container finished" podID="85766866-9614-45d4-a5b3-fba1fcae567d" containerID="c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246" exitCode=0 Oct 30 19:37:33.725159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.725141 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" event={"ID":"85766866-9614-45d4-a5b3-fba1fcae567d","Type":"ContainerDied","Data":"c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246"} Oct 30 19:37:33.725274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.725264 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" event={"ID":"85766866-9614-45d4-a5b3-fba1fcae567d","Type":"ContainerDied","Data":"f94d7dede5e5b3be1c6d4e80f4eb3a6f701e82733798bf92c1576c51e12ede83"} Oct 30 19:37:33.725349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.725340 6514 scope.go:122] "RemoveContainer" containerID="c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246" Oct 30 19:37:33.725615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.725598 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl" Oct 30 19:37:33.734989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.730118 6514 generic.go:356] "Generic (PLEG): container finished" podID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerID="87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f" exitCode=0 Oct 30 19:37:33.734989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.730218 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" event={"ID":"db9eeec3-9f96-483e-ae46-54ff7776dd06","Type":"ContainerDied","Data":"87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f"} Oct 30 19:37:33.734989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.730263 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" event={"ID":"db9eeec3-9f96-483e-ae46-54ff7776dd06","Type":"ContainerDied","Data":"01f4966ea7cb88364538086cb375a27df4fcb5e3c925f446dfda2075d3c58dc4"} Oct 30 19:37:33.734989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.730372 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45" Oct 30 19:37:33.755521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.746240 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" event={"ID":"6119e345-2afa-428d-9cc6-f1d2f9140389","Type":"ContainerStarted","Data":"f62935828af7fd6d1a43a603d68346363c3b6812470ba8e0c8d06cbe9859405c"} Oct 30 19:37:33.770418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.770369 6514 scope.go:122] "RemoveContainer" containerID="c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246" Oct 30 19:37:33.771613 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:33.771426 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246\": container with ID starting with c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246 not found: ID does not exist" containerID="c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246" Oct 30 19:37:33.771613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.771480 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246"} err="failed to get container status \"c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246\": rpc error: code = NotFound desc = could not find container \"c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246\": container with ID starting with c478bffe81d86382dca3086c5581b08bcf82aaa52a3e98f996bbad3c1dd64246 not found: ID does not exist" Oct 30 19:37:33.771613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.771506 6514 scope.go:122] "RemoveContainer" containerID="87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f" Oct 30 19:37:33.776588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.776559 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45"] Oct 30 19:37:33.785052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.785028 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-2ch45"] Oct 30 19:37:33.800712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.800598 6514 scope.go:122] "RemoveContainer" containerID="87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f" Oct 30 19:37:33.801386 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:33.801283 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f\": container with ID starting with 87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f not found: ID does not exist" containerID="87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f" Oct 30 19:37:33.801386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.801325 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f"} err="failed to get container status \"87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f\": rpc error: code = NotFound desc = could not find container \"87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f\": container with ID starting with 87b384b9c7f89aa9814ae2cb0d14afa56f499ab36077fe9963bb6c20f5a3184f not found: ID does not exist" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.806962 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl"] Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807598 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807623 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807639 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d786x\" (UniqueName: \"kubernetes.io/projected/85766866-9614-45d4-a5b3-fba1fcae567d-kube-api-access-d786x\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807653 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807668 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807682 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85766866-9614-45d4-a5b3-fba1fcae567d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.808496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.807695 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85766866-9614-45d4-a5b3-fba1fcae567d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.817207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.815487 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dwtpl"] Oct 30 19:37:33.945402 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:33.945380 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:37:34.013746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.013708 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.013808 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.013856 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.013940 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.014185 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.014221 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.014258 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.014326 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.014386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.014362 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket\") pod \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\" (UID: \"34097a6a-eb26-4df5-aaec-3ad0126a0f40\") " Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.015172 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.015219 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.015295 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.015370 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.015435 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.015365 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.017004 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.017072 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "kube-api-access-vlntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:34.018841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.018657 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:34.019547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.019519 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy" pod "34097a6a-eb26-4df5-aaec-3ad0126a0f40" (UID: "34097a6a-eb26-4df5-aaec-3ad0126a0f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.116874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116839 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116882 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116893 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116903 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116913 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlntw\" (UniqueName: \"kubernetes.io/projected/34097a6a-eb26-4df5-aaec-3ad0126a0f40-kube-api-access-vlntw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116922 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116931 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.117024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.116939 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34097a6a-eb26-4df5-aaec-3ad0126a0f40-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.162582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.162546 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" path="/var/lib/kubelet/pods/85766866-9614-45d4-a5b3-fba1fcae567d/volumes" Oct 30 19:37:34.163309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.163285 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" path="/var/lib/kubelet/pods/db9eeec3-9f96-483e-ae46-54ff7776dd06/volumes" Oct 30 19:37:34.456723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.456693 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:34.603161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.603115 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:34.607955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.607915 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" containerID="cri-o://d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026" gracePeriod=25 Oct 30 19:37:34.632741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.630608 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerName="payload-pre-processing" containerID="cri-o://eb387434a26e500a04e9ffbc11b5737d77be559f84898e3d0a101d97af5e7a33" gracePeriod=25 Oct 30 19:37:34.774260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.774081 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.825613 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" event={"ID":"6119e345-2afa-428d-9cc6-f1d2f9140389","Type":"ContainerStarted","Data":"bf04d3e879d7889c9bfa3c322d582b6abe0c7d5c9bf824d59c2800d07d8c5a52"} Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.827401 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.831297 6514 generic.go:356] "Generic (PLEG): container finished" podID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerID="9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0" exitCode=0 Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.831343 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.831438 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" event={"ID":"34097a6a-eb26-4df5-aaec-3ad0126a0f40","Type":"ContainerDied","Data":"9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0"} Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.831476 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm" event={"ID":"34097a6a-eb26-4df5-aaec-3ad0126a0f40","Type":"ContainerDied","Data":"bbbae329d7999be4c217e59a84b31d3586f23b2a6f03ec312670c1837a1c8194"} Oct 30 19:37:34.833040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.831503 6514 scope.go:122] "RemoveContainer" containerID="9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0" Oct 30 19:37:34.861350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.861297 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" podStartSLOduration=2.376497037 podStartE2EDuration="2.86128009s" totalImagesPullingTime="484.783053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:34.854617335 +0000 UTC m=+1224.812436524" watchObservedRunningTime="2026-10-30 19:37:34.86128009 +0000 UTC m=+1224.819099305" Oct 30 19:37:34.867142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.867119 6514 scope.go:122] "RemoveContainer" containerID="9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0" Oct 30 19:37:34.867511 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:34.867486 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0\": container with ID starting with 9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0 not found: ID does not exist" containerID="9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0" Oct 30 19:37:34.867578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.867524 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0"} err="failed to get container status \"9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0\": rpc error: code = NotFound desc = could not find container \"9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0\": container with ID starting with 9831e2d2bdde0cc7d2beac66ee50c5b8967260e69f3e9620b36aa66cca6d99c0 not found: ID does not exist" Oct 30 19:37:34.885783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.885738 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm"] Oct 30 19:37:34.940654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.938573 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:34.942827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:34.942628 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-d6kpm"] Oct 30 19:37:35.327794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:35.327763 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:37:35.661218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:35.661133 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:35.802029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:35.801443 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:36.201886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:36.200637 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:36.240126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:36.240089 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" path="/var/lib/kubelet/pods/34097a6a-eb26-4df5-aaec-3ad0126a0f40/volumes" Oct 30 19:37:36.359900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:36.359003 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:37:36.383888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:36.380219 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:36.883230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:36.882015 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:37.665572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:37.664104 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:37.665572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:37.664892 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:37:38.540888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:38.540072 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:39.345494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.345445 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" containerID="cri-o://cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2" gracePeriod=45 Oct 30 19:37:39.532888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.525320 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:39.809051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.809025 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:37:39.884913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.884850 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs\" (UniqueName: \"kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs\") pod \"46c3202a-06f7-4864-8039-376c19c089be\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " Oct 30 19:37:39.885080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.884971 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home\") pod \"46c3202a-06f7-4864-8039-376c19c089be\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " Oct 30 19:37:39.885275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.885241 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home" pod "46c3202a-06f7-4864-8039-376c19c089be" (UID: "46c3202a-06f7-4864-8039-376c19c089be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.885448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.885427 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache\") pod \"46c3202a-06f7-4864-8039-376c19c089be\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " Oct 30 19:37:39.885697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.885651 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache" pod "46c3202a-06f7-4864-8039-376c19c089be" (UID: "46c3202a-06f7-4864-8039-376c19c089be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.885885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.885845 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95\") pod \"46c3202a-06f7-4864-8039-376c19c089be\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " Oct 30 19:37:39.886563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.886542 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir\") pod \"46c3202a-06f7-4864-8039-376c19c089be\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " Oct 30 19:37:39.887084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.887064 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm\") pod \"46c3202a-06f7-4864-8039-376c19c089be\" (UID: \"46c3202a-06f7-4864-8039-376c19c089be\") " Oct 30 19:37:39.887359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.886941 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir" pod "46c3202a-06f7-4864-8039-376c19c089be" (UID: "46c3202a-06f7-4864-8039-376c19c089be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.887760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.887738 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs" pod "46c3202a-06f7-4864-8039-376c19c089be" (UID: "46c3202a-06f7-4864-8039-376c19c089be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:39.888245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.888192 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.888375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.888350 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.888453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.888376 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c3202a-06f7-4864-8039-376c19c089be-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.888453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.888390 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.889081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.889058 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95" pod "46c3202a-06f7-4864-8039-376c19c089be" (UID: "46c3202a-06f7-4864-8039-376c19c089be"). InnerVolumeSpecName "kube-api-access-4wq95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:39.889536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.889511 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm" pod "46c3202a-06f7-4864-8039-376c19c089be" (UID: "46c3202a-06f7-4864-8039-376c19c089be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.917642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.917611 6514 generic.go:356] "Generic (PLEG): container finished" podID="46c3202a-06f7-4864-8039-376c19c089be" containerID="cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2" exitCode=0 Oct 30 19:37:39.917745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.917661 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" Oct 30 19:37:39.917745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.917679 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" event={"ID":"46c3202a-06f7-4864-8039-376c19c089be","Type":"ContainerDied","Data":"cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2"} Oct 30 19:37:39.917745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.917730 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" event={"ID":"46c3202a-06f7-4864-8039-376c19c089be","Type":"ContainerDied","Data":"7681dc3720d35d118b31e46bce3d029cb712579537181388cef17f3377448f4f"} Oct 30 19:37:39.917895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.917752 6514 scope.go:122] "RemoveContainer" containerID="cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2" Oct 30 19:37:39.929301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.929281 6514 scope.go:122] "RemoveContainer" containerID="cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2" Oct 30 19:37:39.929622 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:39.929597 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2\": container with ID starting with cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2 not found: ID does not exist" containerID="cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2" Oct 30 19:37:39.929684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.929634 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2"} err="failed to get container status \"cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2\": rpc error: code = NotFound desc = could not find container \"cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2\": container with ID starting with cfc6fe2f49b6aca44028904786eb8870d1388131d0fd2d30ee2e0a64411364e2 not found: ID does not exist" Oct 30 19:37:39.959956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.959924 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb"] Oct 30 19:37:39.963704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.963683 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb"] Oct 30 19:37:39.989635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.989610 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/46c3202a-06f7-4864-8039-376c19c089be-kube-api-access-4wq95\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.989635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:39.989636 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c3202a-06f7-4864-8039-376c19c089be-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:40.166314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:40.166227 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c3202a-06f7-4864-8039-376c19c089be" path="/var/lib/kubelet/pods/46c3202a-06f7-4864-8039-376c19c089be/volumes" Oct 30 19:37:40.375484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:40.375447 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:40.716801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:40.716769 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:40.790898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:40.784529 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/simulator-w5-60577a-kserve-7cfd9755b6-555fb" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:41.380988 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:41.380945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:41.616963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:41.616933 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:41.780632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:41.780597 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:41.950497 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:41.950465 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:37:42.258514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:42.258478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:42.501100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:42.501063 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw"] Oct 30 19:37:42.501614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:42.501267 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" podUID="ad8e8a4c-2fbc-4597-80bb-e72412918855" containerName="maas-api" containerID="cri-o://b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36" gracePeriod=30 Oct 30 19:37:42.567272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:42.567242 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:43.095487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.095455 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:37:43.119386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.119338 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:43.157014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.156984 6514 generic.go:356] "Generic (PLEG): container finished" podID="ad8e8a4c-2fbc-4597-80bb-e72412918855" containerID="b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36" exitCode=0 Oct 30 19:37:43.157202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.157020 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" Oct 30 19:37:43.157202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.157061 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" event={"ID":"ad8e8a4c-2fbc-4597-80bb-e72412918855","Type":"ContainerDied","Data":"b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36"} Oct 30 19:37:43.157202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.157106 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw" event={"ID":"ad8e8a4c-2fbc-4597-80bb-e72412918855","Type":"ContainerDied","Data":"d453e1e2bc6441bd4c2fb38d2ee111df8e4131d01c4c4cf50816922f86e416d2"} Oct 30 19:37:43.157202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.157156 6514 scope.go:122] "RemoveContainer" containerID="b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36" Oct 30 19:37:43.172129 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.172102 6514 scope.go:122] "RemoveContainer" containerID="b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36" Oct 30 19:37:43.173034 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:43.173003 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36\": container with ID starting with b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36 not found: ID does not exist" containerID="b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36" Oct 30 19:37:43.173208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.173044 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36"} err="failed to get container status \"b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36\": rpc error: code = NotFound desc = could not find container \"b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36\": container with ID starting with b01b3d64d4cc0311449f4dcfdc86c0c3c1b1d0de8e558d3d35fc2058031c4c36 not found: ID does not exist" Oct 30 19:37:43.214934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.214880 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs\") pod \"ad8e8a4c-2fbc-4597-80bb-e72412918855\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " Oct 30 19:37:43.215105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.215017 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca\") pod \"ad8e8a4c-2fbc-4597-80bb-e72412918855\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " Oct 30 19:37:43.215186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.215108 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\") pod \"ad8e8a4c-2fbc-4597-80bb-e72412918855\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " Oct 30 19:37:43.215253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.215210 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5\" (UniqueName: \"kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5\") pod \"ad8e8a4c-2fbc-4597-80bb-e72412918855\" (UID: \"ad8e8a4c-2fbc-4597-80bb-e72412918855\") " Oct 30 19:37:43.215407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.215380 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca" pod "ad8e8a4c-2fbc-4597-80bb-e72412918855" (UID: "ad8e8a4c-2fbc-4597-80bb-e72412918855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:43.216557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.216532 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8e8a4c-2fbc-4597-80bb-e72412918855-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:43.217190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.217163 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs" pod "ad8e8a4c-2fbc-4597-80bb-e72412918855" (UID: "ad8e8a4c-2fbc-4597-80bb-e72412918855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:43.217480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.217454 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5" pod "ad8e8a4c-2fbc-4597-80bb-e72412918855" (UID: "ad8e8a4c-2fbc-4597-80bb-e72412918855"). InnerVolumeSpecName "kube-api-access-xcfs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:43.217686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.217670 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls" pod "ad8e8a4c-2fbc-4597-80bb-e72412918855" (UID: "ad8e8a4c-2fbc-4597-80bb-e72412918855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:43.236751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.236726 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:43.322547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.319505 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfs5\" (UniqueName: \"kubernetes.io/projected/ad8e8a4c-2fbc-4597-80bb-e72412918855-kube-api-access-xcfs5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:43.322547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.319546 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:43.322547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.319561 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8e8a4c-2fbc-4597-80bb-e72412918855-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:43.374899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.364748 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:43.483709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.483677 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw"] Oct 30 19:37:43.488018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.487991 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73f96d-7488969884-vgffw"] Oct 30 19:37:43.741544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.741502 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:43.855816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:43.855757 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:44.162468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:44.162390 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8e8a4c-2fbc-4597-80bb-e72412918855" path="/var/lib/kubelet/pods/ad8e8a4c-2fbc-4597-80bb-e72412918855/volumes" Oct 30 19:37:44.216446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:44.216409 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:45.329226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:45.329176 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:37:45.351745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:45.351713 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:46.126954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.126916 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784"] Oct 30 19:37:46.127927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.127902 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ad8e8a4c-2fbc-4597-80bb-e72412918855" containerName="maas-api" Oct 30 19:37:46.127927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.127923 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8e8a4c-2fbc-4597-80bb-e72412918855" containerName="maas-api" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.127940 6514 cpu_manager.go:414] "containerMap: removing container" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.127945 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.127979 6514 cpu_manager.go:414] "containerMap: removing container" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.127985 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128011 6514 cpu_manager.go:414] "containerMap: removing container" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128016 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128028 6514 cpu_manager.go:414] "containerMap: removing container" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128033 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128144 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c3202a-06f7-4864-8039-376c19c089be" containerName="main" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128154 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9eeec3-9f96-483e-ae46-54ff7776dd06" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128165 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="85766866-9614-45d4-a5b3-fba1fcae567d" containerName="istio-proxy" Oct 30 19:37:46.128166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128173 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="34097a6a-eb26-4df5-aaec-3ad0126a0f40" containerName="istio-proxy" Oct 30 19:37:46.128812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.128184 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8e8a4c-2fbc-4597-80bb-e72412918855" containerName="maas-api" Oct 30 19:37:46.133880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.133850 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.145569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.145548 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784"] Oct 30 19:37:46.178143 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.178112 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth"] Oct 30 19:37:46.187013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.186989 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.197126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.197099 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth"] Oct 30 19:37:46.299243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299210 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.299418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299294 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.299494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299443 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.299553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299533 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ld2\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.299656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299630 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.299787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299763 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.299835 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.299809 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.300017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300000 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.300066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300039 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.300113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300096 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.300164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300142 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.300270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300250 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.300328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300293 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc87m\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.300422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300398 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.300474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300458 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.300551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300535 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.300624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300609 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.300656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.300638 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.311411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.311383 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:46.365420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.364967 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:37:46.406350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.406210 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.406636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.406615 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ld2\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.406762 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.406748 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.406959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.406941 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.407095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407078 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.407284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407270 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.407388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407374 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.407486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407473 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.407586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407574 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.407716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407704 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.408022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.407997 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc87m\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.408181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.408163 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.408282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.408269 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.408428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.408415 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.408561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.408548 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.408651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.408637 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.416390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.416362 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.416674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.416652 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.416978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.414183 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.431933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.414795 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.431933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.415038 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.416021 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.424113 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.427126 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.427268 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.427567 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.410419 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.432133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.410794 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.432547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.432486 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.433418 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.433392 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.433924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.433848 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.435377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.435355 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.435690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.435666 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc87m\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.435930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.435909 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.436294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.436273 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ld2\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.438205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.438158 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.452019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.451997 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:46.510344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.510182 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:46.741070 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:37:46.741039 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abc9e45_7234_4330_962b_de93f1ce7a94.slice/crio-0e0c1450ebceb966a26bb77fa223e17ab9e731374349c84de804216c68a1be81 WatchSource:0}: Error finding container 0e0c1450ebceb966a26bb77fa223e17ab9e731374349c84de804216c68a1be81: Status 404 returned error can't find the container with id 0e0c1450ebceb966a26bb77fa223e17ab9e731374349c84de804216c68a1be81 Oct 30 19:37:46.741841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.741814 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784"] Oct 30 19:37:46.744300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.744268 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:46.744398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.744349 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:46.744446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.744395 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:46.825108 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:37:46.825072 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b4b94b_944f_4382_9f7f_c8614d1b5de3.slice/crio-5d1b7dabad9b5960293b4e1a400df167fcfcd890b063766453a357ebd16d4b7a WatchSource:0}: Error finding container 5d1b7dabad9b5960293b4e1a400df167fcfcd890b063766453a357ebd16d4b7a: Status 404 returned error can't find the container with id 5d1b7dabad9b5960293b4e1a400df167fcfcd890b063766453a357ebd16d4b7a Oct 30 19:37:46.825982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.825955 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth"] Oct 30 19:37:46.828057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.828028 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:46.828137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.828089 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:46.828137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:46.828117 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:47.209673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.209633 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:47.221888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.219098 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" event={"ID":"16b4b94b-944f-4382-9f7f-c8614d1b5de3","Type":"ContainerStarted","Data":"1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7"} Oct 30 19:37:47.221888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.219183 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" event={"ID":"16b4b94b-944f-4382-9f7f-c8614d1b5de3","Type":"ContainerStarted","Data":"5d1b7dabad9b5960293b4e1a400df167fcfcd890b063766453a357ebd16d4b7a"} Oct 30 19:37:47.241996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.239789 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podStartSLOduration=1.239771597 podStartE2EDuration="1.239771597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:47.23938618 +0000 UTC m=+1237.197205370" watchObservedRunningTime="2026-10-30 19:37:47.239771597 +0000 UTC m=+1237.197590788" Oct 30 19:37:47.253110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.251998 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" event={"ID":"4abc9e45-7234-4330-962b-de93f1ce7a94","Type":"ContainerStarted","Data":"04055d1ed9fe76e6c0d7c15dca4a15785d69e1c61f6916ce454715bf55ff3986"} Oct 30 19:37:47.253110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.252100 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" event={"ID":"4abc9e45-7234-4330-962b-de93f1ce7a94","Type":"ContainerStarted","Data":"0e0c1450ebceb966a26bb77fa223e17ab9e731374349c84de804216c68a1be81"} Oct 30 19:37:47.303169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.299810 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podStartSLOduration=1.299781755 podStartE2EDuration="1.299781755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:47.292249124 +0000 UTC m=+1237.250068313" watchObservedRunningTime="2026-10-30 19:37:47.299781755 +0000 UTC m=+1237.257600943" Oct 30 19:37:47.452691 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.452658 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:37:47.454544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.454514 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:37:47.454679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.454587 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:37:47.512421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.512323 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:37:47.516049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.515928 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:37:47.516419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:47.516294 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:37:48.452396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:48.452356 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:37:48.452574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:48.452419 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:37:48.463337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:48.463303 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:48.516056 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:48.516016 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:37:48.516354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:48.516321 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:37:49.076415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.076374 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:49.452909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.452364 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:37:49.452909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.452413 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:37:49.519885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.519332 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:37:49.519885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.519395 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:37:49.768646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.768523 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:49.881184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:49.881130 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:50.486890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:50.444150 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:50.486890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:50.453214 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:37:50.486890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:50.453276 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:37:50.516898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:50.513473 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:37:50.533006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:50.532947 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:37:50.678886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:50.641386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:50.976994 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:50.976957 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:37:51.456287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.456246 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:37:51.456599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.456567 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:37:51.523287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.523248 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:37:51.523539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.523511 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:37:51.539997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.539959 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:51.656293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.656261 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:51.917945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:51.895421 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:52.357829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.357795 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:52.472054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.467849 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:37:52.472054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.467937 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:37:52.519532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.515336 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:52.519532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.515838 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:37:52.519532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.515908 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:37:52.672943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.668766 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784"] Oct 30 19:37:52.673516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.673495 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth"] Oct 30 19:37:52.673888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.673835 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" containerID="cri-o://1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7" gracePeriod=30 Oct 30 19:37:52.674295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.674271 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" containerID="cri-o://04055d1ed9fe76e6c0d7c15dca4a15785d69e1c61f6916ce454715bf55ff3986" gracePeriod=30 Oct 30 19:37:52.674522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.674377 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg"] Oct 30 19:37:52.674948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.674925 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:52.787808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:52.786746 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" containerID="cri-o://ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e" gracePeriod=30 Oct 30 19:37:53.870671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:53.870630 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:37:54.016891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:54.016481 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:54.183936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:54.182553 6514 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:37:54.183936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:54.182609 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:37:54.334531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:54.334422 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:54.628397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:54.628313 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:55.468849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.468754 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:37:55.567478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.567432 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:55.570588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.568960 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume\") pod \"6b2d179b-367b-4a85-9044-e9edb74fd53b\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " Oct 30 19:37:55.571155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.571127 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q\" (UniqueName: \"kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q\") pod \"6b2d179b-367b-4a85-9044-e9edb74fd53b\" (UID: \"6b2d179b-367b-4a85-9044-e9edb74fd53b\") " Oct 30 19:37:55.574291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.570472 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume" pod "6b2d179b-367b-4a85-9044-e9edb74fd53b" (UID: "6b2d179b-367b-4a85-9044-e9edb74fd53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:55.574581 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.574530 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q" pod "6b2d179b-367b-4a85-9044-e9edb74fd53b" (UID: "6b2d179b-367b-4a85-9044-e9edb74fd53b"). InnerVolumeSpecName "kube-api-access-tsb8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:55.575382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.575361 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsb8q\" (UniqueName: \"kubernetes.io/projected/6b2d179b-367b-4a85-9044-e9edb74fd53b-kube-api-access-tsb8q\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:55.575574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.575557 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2d179b-367b-4a85-9044-e9edb74fd53b-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:55.728980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.708179 6514 generic.go:356] "Generic (PLEG): container finished" podID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerID="d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026" exitCode=0 Oct 30 19:37:55.728980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.708401 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" event={"ID":"6b2d179b-367b-4a85-9044-e9edb74fd53b","Type":"ContainerDied","Data":"d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026"} Oct 30 19:37:55.728980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.708451 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" event={"ID":"6b2d179b-367b-4a85-9044-e9edb74fd53b","Type":"ContainerDied","Data":"1f38f9f640ade649bbe8f6f6b4547f67ef7fc50aedb972e17be8ccee5c760271"} Oct 30 19:37:55.728980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.708482 6514 scope.go:122] "RemoveContainer" containerID="d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026" Oct 30 19:37:55.728980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.708631 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" Oct 30 19:37:55.750601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.748905 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:55.752684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.752657 6514 generic.go:356] "Generic (PLEG): container finished" podID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerID="eb387434a26e500a04e9ffbc11b5737d77be559f84898e3d0a101d97af5e7a33" exitCode=0 Oct 30 19:37:55.753941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.752998 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" event={"ID":"0fd3059d-f277-495c-b6a9-8fea7f88eb2f","Type":"ContainerDied","Data":"eb387434a26e500a04e9ffbc11b5737d77be559f84898e3d0a101d97af5e7a33"} Oct 30 19:37:55.754180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.754151 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" event={"ID":"0fd3059d-f277-495c-b6a9-8fea7f88eb2f","Type":"ContainerDied","Data":"38216830188d786eb495111bb38949b9640bd375cbf5d99b7a01a39209a25609"} Oct 30 19:37:55.754352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.754331 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38216830188d786eb495111bb38949b9640bd375cbf5d99b7a01a39209a25609" Oct 30 19:37:55.764026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.763946 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:37:55.786220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.786198 6514 scope.go:122] "RemoveContainer" containerID="d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026" Oct 30 19:37:55.787201 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:55.787176 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026\": container with ID starting with d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026 not found: ID does not exist" containerID="d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026" Oct 30 19:37:55.787364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.787337 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026"} err="failed to get container status \"d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026\": rpc error: code = NotFound desc = could not find container \"d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026\": container with ID starting with d0c4dbf6aedac8c0aeb058b0825af91be07799b0cea0f9e5959e73f8523f3026 not found: ID does not exist" Oct 30 19:37:55.832107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.832066 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn\" (UniqueName: \"kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn\") pod \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " Oct 30 19:37:55.832314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.832189 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume\") pod \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\" (UID: \"0fd3059d-f277-495c-b6a9-8fea7f88eb2f\") " Oct 30 19:37:55.833477 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.833442 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume" pod "0fd3059d-f277-495c-b6a9-8fea7f88eb2f" (UID: "0fd3059d-f277-495c-b6a9-8fea7f88eb2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:55.852610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.852562 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn" pod "0fd3059d-f277-495c-b6a9-8fea7f88eb2f" (UID: "0fd3059d-f277-495c-b6a9-8fea7f88eb2f"). InnerVolumeSpecName "kube-api-access-s42fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:55.860134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.860110 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j"] Oct 30 19:37:55.870328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.866810 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j"] Oct 30 19:37:55.946503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.946462 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42fn\" (UniqueName: \"kubernetes.io/projected/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-kube-api-access-s42fn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:55.946753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.946733 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd3059d-f277-495c-b6a9-8fea7f88eb2f-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:55.972920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:55.971925 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:56.249883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.232673 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:56.369912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.366469 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-60577a-646d79c448-2w78j" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded" Oct 30 19:37:56.369912 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:37:56.366946 6514 logging.go:55] [core] [Channel #3292 SubChannel #3293]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.45:50052: operation was canceled" Oct 30 19:37:56.410427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.409685 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" path="/var/lib/kubelet/pods/6b2d179b-367b-4a85-9044-e9edb74fd53b/volumes" Oct 30 19:37:56.470885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.470808 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:56.617939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.615255 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:56.761706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.761677 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs" Oct 30 19:37:56.786844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.786812 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs"] Oct 30 19:37:56.822134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:56.822098 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-60577a-785b689d99-wh2hs"] Oct 30 19:37:57.018893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.014402 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:57.316963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.312285 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:57.444421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.444388 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89"] Oct 30 19:37:57.445345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.445321 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerName="payload-pre-processing" Oct 30 19:37:57.445345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.445344 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerName="payload-pre-processing" Oct 30 19:37:57.445537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.445370 6514 cpu_manager.go:414] "containerMap: removing container" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" Oct 30 19:37:57.445537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.445379 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" Oct 30 19:37:57.445640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.445556 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" containerName="payload-pre-processing" Oct 30 19:37:57.445640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.445572 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2d179b-367b-4a85-9044-e9edb74fd53b" containerName="payload-processing" Oct 30 19:37:57.451010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.450990 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.463892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.463848 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89"] Oct 30 19:37:57.470209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.470187 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d651c4-openshift-default-dockercfg-gxzm6\"" Oct 30 19:37:57.534079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534039 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534274 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534394 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqcg\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534464 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534490 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534531 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534674 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534705 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.534878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.534855 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.607268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.607179 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.646846 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.647552 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651092 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651226 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqcg\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651283 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651309 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651340 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651407 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651432 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.651643 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.652178 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.652676 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.656888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.652938 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.681888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.675073 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.695888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.695785 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.709888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.703623 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.764028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.737486 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqcg\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.776976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.776530 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token\") pod \"e2e-inf-b-d651c4-openshift-default-775869444c-9wh89\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.796887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.785278 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:37:57.805897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:57.802374 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:58.282903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.282875 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd3059d-f277-495c-b6a9-8fea7f88eb2f" path="/var/lib/kubelet/pods/0fd3059d-f277-495c-b6a9-8fea7f88eb2f/volumes" Oct 30 19:37:58.484518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.483333 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89"] Oct 30 19:37:58.485191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.485156 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:58.485309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.485225 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:58.485309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.485260 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:58.630089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.630056 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:37:58.654515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.653627 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.654515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.653743 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.654515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.653812 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.656431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.656404 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.657730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.657704 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.657967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.657949 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.658103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.658088 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.658236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.658223 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.660489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.660456 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy\") pod \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\" (UID: \"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a\") " Oct 30 19:37:58.678810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.678765 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.679625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.679596 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.680291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.680266 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:58.689342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.689292 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "kube-api-access-nzkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.689685 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.689647 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.694975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.694938 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.695294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.695271 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:58.699248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.692187 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.761147 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.754156 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token" pod "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" (UID: "886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771195 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771226 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771242 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771258 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771273 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771287 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771303 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzkgd\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-kube-api-access-nzkgd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771319 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.781202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.771333 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.813397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.804341 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:58.971877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.971734 6514 generic.go:356] "Generic (PLEG): container finished" podID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerID="ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e" exitCode=0 Oct 30 19:37:58.976336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.972089 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" event={"ID":"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a","Type":"ContainerDied","Data":"ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e"} Oct 30 19:37:58.976336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.972228 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" event={"ID":"886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a","Type":"ContainerDied","Data":"94679fedf62363609a1f0de0c5c207f0664d3a443f00a81fec22714ef30a2792"} Oct 30 19:37:58.976336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.972263 6514 scope.go:122] "RemoveContainer" containerID="ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e" Oct 30 19:37:58.976336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.972441 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg" Oct 30 19:37:58.981701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:58.981617 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" event={"ID":"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7","Type":"ContainerStarted","Data":"483aad3f2b958cfc10a916a0f3a5d3a2fcfe12a883677f9b3482b9e7a3afeba2"} Oct 30 19:37:59.044995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:59.044911 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg"] Oct 30 19:37:59.053577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:59.053443 6514 scope.go:122] "RemoveContainer" containerID="ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e" Oct 30 19:37:59.054276 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:37:59.054184 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e\": container with ID starting with ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e not found: ID does not exist" containerID="ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e" Oct 30 19:37:59.054276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:59.054220 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e"} err="failed to get container status \"ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e\": rpc error: code = NotFound desc = could not find container \"ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e\": container with ID starting with ec78bf4ef0cd1cabf1cc37912c6bcffbac58eb765f86a7be1c18372b6917650e not found: ID does not exist" Oct 30 19:37:59.064131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:37:59.064108 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-pgfhg"] Oct 30 19:38:00.024502 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.024449 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" event={"ID":"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7","Type":"ContainerStarted","Data":"9ef9b1224be0ac04cb6c298089bd3fd1b3fb430e3653b53f41f0c71ed16be8c9"} Oct 30 19:38:00.098977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.098933 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:00.130946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.130504 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podStartSLOduration=3.130482228 podStartE2EDuration="3.130482228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:00.086316348 +0000 UTC m=+1250.044135538" watchObservedRunningTime="2026-10-30 19:38:00.130482228 +0000 UTC m=+1250.088301418" Oct 30 19:38:00.415449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.415363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:00.438964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.438935 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" path="/var/lib/kubelet/pods/886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a/volumes" Oct 30 19:38:00.542739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.542692 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:00.786174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.786136 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:38:00.812256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:00.812216 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:38:01.039798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.039691 6514 generic.go:356] "Generic (PLEG): container finished" podID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerID="04055d1ed9fe76e6c0d7c15dca4a15785d69e1c61f6916ce454715bf55ff3986" exitCode=0 Oct 30 19:38:01.040979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.040932 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" event={"ID":"4abc9e45-7234-4330-962b-de93f1ce7a94","Type":"ContainerDied","Data":"04055d1ed9fe76e6c0d7c15dca4a15785d69e1c61f6916ce454715bf55ff3986"} Oct 30 19:38:01.041167 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.040996 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:38:01.041889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.041816 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:38:01.246679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.246649 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:38:01.436388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.429288 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:01.464890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.444600 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.465225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.465192 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.465390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.465375 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.465573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.465559 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.465747 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.465726 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.469907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.465921 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.470657 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.470759 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.470853 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo\") pod \"4abc9e45-7234-4330-962b-de93f1ce7a94\" (UID: \"4abc9e45-7234-4330-962b-de93f1ce7a94\") " Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.503248 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.503470 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.503628 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.520474 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:01.535886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.521991 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:38:01.540797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.537983 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:01.540797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.538303 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:01.540797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.539480 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:01.540797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.539657 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "kube-api-access-v6ld2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:01.540797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.540744 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data" pod "4abc9e45-7234-4330-962b-de93f1ce7a94" (UID: "4abc9e45-7234-4330-962b-de93f1ce7a94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:01.573807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573763 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ld2\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-kube-api-access-v6ld2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.573807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573796 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.573807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573806 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.573807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573816 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.574164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573825 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.574164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573836 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.574164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573844 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc9e45-7234-4330-962b-de93f1ce7a94-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.574164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573853 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc9e45-7234-4330-962b-de93f1ce7a94-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.574164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.573898 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc9e45-7234-4330-962b-de93f1ce7a94-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:01.857806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.857694 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:01.903227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.903196 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:38:01.995631 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.995602 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.995899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.995878 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996104 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996416 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996597 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996573 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996687 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996671 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996704 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996760 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:01.996931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:01.996799 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m\") pod \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\" (UID: \"16b4b94b-944f-4382-9f7f-c8614d1b5de3\") " Oct 30 19:38:02.033917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.020088 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:02.033917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.020422 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:02.033917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.025953 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:02.041281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.041091 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:02.042219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.042182 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125502 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125670 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125696 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125711 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125724 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125736 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.136886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.125750 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:02.232910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.232697 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.315377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.315285 6514 generic.go:356] "Generic (PLEG): container finished" podID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerID="1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7" exitCode=0 Oct 30 19:38:02.315522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.315479 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" Oct 30 19:38:02.327912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.327502 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:02.329071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.329038 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:02.329205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.329151 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m" pod "16b4b94b-944f-4382-9f7f-c8614d1b5de3" (UID: "16b4b94b-944f-4382-9f7f-c8614d1b5de3"). InnerVolumeSpecName "kube-api-access-rc87m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:02.334572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.334548 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.334649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.334578 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4b94b-944f-4382-9f7f-c8614d1b5de3-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.334649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.334611 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc87m\" (UniqueName: \"kubernetes.io/projected/16b4b94b-944f-4382-9f7f-c8614d1b5de3-kube-api-access-rc87m\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:02.336337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.336312 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" Oct 30 19:38:02.338245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.338219 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" event={"ID":"16b4b94b-944f-4382-9f7f-c8614d1b5de3","Type":"ContainerDied","Data":"1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7"} Oct 30 19:38:02.338319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.338274 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth" event={"ID":"16b4b94b-944f-4382-9f7f-c8614d1b5de3","Type":"ContainerDied","Data":"5d1b7dabad9b5960293b4e1a400df167fcfcd890b063766453a357ebd16d4b7a"} Oct 30 19:38:02.338319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.338300 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784" event={"ID":"4abc9e45-7234-4330-962b-de93f1ce7a94","Type":"ContainerDied","Data":"0e0c1450ebceb966a26bb77fa223e17ab9e731374349c84de804216c68a1be81"} Oct 30 19:38:02.338383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.338332 6514 scope.go:122] "RemoveContainer" containerID="1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7" Oct 30 19:38:02.353164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.353069 6514 scope.go:122] "RemoveContainer" containerID="1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7" Oct 30 19:38:02.353396 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:02.353365 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7\": container with ID starting with 1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7 not found: ID does not exist" containerID="1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7" Oct 30 19:38:02.353505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.353407 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7"} err="failed to get container status \"1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7\": rpc error: code = NotFound desc = could not find container \"1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7\": container with ID starting with 1d8231ef07aa9358b0fc164dc113ee2779b04209861e5cad2bf5e977e78af5b7 not found: ID does not exist" Oct 30 19:38:02.353505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.353441 6514 scope.go:122] "RemoveContainer" containerID="04055d1ed9fe76e6c0d7c15dca4a15785d69e1c61f6916ce454715bf55ff3986" Oct 30 19:38:02.369802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.369780 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784"] Oct 30 19:38:02.375668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.375640 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-7f784"] Oct 30 19:38:02.650436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.650401 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.659411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.659385 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth"] Oct 30 19:38:02.685960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.685926 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-xbjth"] Oct 30 19:38:02.692711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.692683 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj"] Oct 30 19:38:02.692965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.692940 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" podUID="8930e474-eb64-4d26-abe4-45d6f1d9c0df" containerName="maas-api" containerID="cri-o://67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b" gracePeriod=30 Oct 30 19:38:02.822338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.822299 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.971547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.971517 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.997579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:02.997553 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:38:03.045920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.045889 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca\") pod \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " Oct 30 19:38:03.046400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.046074 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs\") pod \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " Oct 30 19:38:03.046400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.046134 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\") pod \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " Oct 30 19:38:03.046400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.046156 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp\" (UniqueName: \"kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp\") pod \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\" (UID: \"8930e474-eb64-4d26-abe4-45d6f1d9c0df\") " Oct 30 19:38:03.046400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.046324 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca" pod "8930e474-eb64-4d26-abe4-45d6f1d9c0df" (UID: "8930e474-eb64-4d26-abe4-45d6f1d9c0df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:03.047135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.047108 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8930e474-eb64-4d26-abe4-45d6f1d9c0df-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.048475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.048448 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls" pod "8930e474-eb64-4d26-abe4-45d6f1d9c0df" (UID: "8930e474-eb64-4d26-abe4-45d6f1d9c0df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:03.048586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.048476 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs" pod "8930e474-eb64-4d26-abe4-45d6f1d9c0df" (UID: "8930e474-eb64-4d26-abe4-45d6f1d9c0df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:03.048586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.048482 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp" pod "8930e474-eb64-4d26-abe4-45d6f1d9c0df" (UID: "8930e474-eb64-4d26-abe4-45d6f1d9c0df"). InnerVolumeSpecName "kube-api-access-pc7dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:03.158014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.149676 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.158014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.149720 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8930e474-eb64-4d26-abe4-45d6f1d9c0df-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.158014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.149736 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc7dp\" (UniqueName: \"kubernetes.io/projected/8930e474-eb64-4d26-abe4-45d6f1d9c0df-kube-api-access-pc7dp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.309504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.309469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:03.439822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.439787 6514 generic.go:356] "Generic (PLEG): container finished" podID="8930e474-eb64-4d26-abe4-45d6f1d9c0df" containerID="67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b" exitCode=0 Oct 30 19:38:03.440942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.440842 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" Oct 30 19:38:03.446215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.446181 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" event={"ID":"8930e474-eb64-4d26-abe4-45d6f1d9c0df","Type":"ContainerDied","Data":"67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b"} Oct 30 19:38:03.446955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.446927 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj" event={"ID":"8930e474-eb64-4d26-abe4-45d6f1d9c0df","Type":"ContainerDied","Data":"5a545a263722c8c5ede2951e10e62ca0fccd47aecc63ce543c0facedd38979c8"} Oct 30 19:38:03.447122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.446973 6514 scope.go:122] "RemoveContainer" containerID="67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b" Oct 30 19:38:03.478517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.477911 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:03.506110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.505594 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj"] Oct 30 19:38:03.523079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.521133 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-60577a-6f5cbd5d94-zx9cj"] Oct 30 19:38:03.523980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.523953 6514 scope.go:122] "RemoveContainer" containerID="67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b" Oct 30 19:38:03.524833 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:03.524802 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b\": container with ID starting with 67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b not found: ID does not exist" containerID="67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b" Oct 30 19:38:03.524956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.524846 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b"} err="failed to get container status \"67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b\": rpc error: code = NotFound desc = could not find container \"67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b\": container with ID starting with 67e849e691840dc4e65d7233463ec8f10600aa63a0ea94b9760ad9f09868c61b not found: ID does not exist" Oct 30 19:38:03.653967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:03.653468 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.166282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.166248 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" path="/var/lib/kubelet/pods/16b4b94b-944f-4382-9f7f-c8614d1b5de3/volumes" Oct 30 19:38:04.167023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.167001 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" path="/var/lib/kubelet/pods/4abc9e45-7234-4330-962b-de93f1ce7a94/volumes" Oct 30 19:38:04.167598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.167580 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8930e474-eb64-4d26-abe4-45d6f1d9c0df" path="/var/lib/kubelet/pods/8930e474-eb64-4d26-abe4-45d6f1d9c0df/volumes" Oct 30 19:38:04.170997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.170967 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.677917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.675538 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.706744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.705935 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv"] Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719395 6514 cpu_manager.go:414] "containerMap: removing container" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719447 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719476 6514 cpu_manager.go:414] "containerMap: removing container" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719489 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719520 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719528 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719559 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8930e474-eb64-4d26-abe4-45d6f1d9c0df" containerName="maas-api" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719566 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8930e474-eb64-4d26-abe4-45d6f1d9c0df" containerName="maas-api" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719792 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="886c3ef7-0f8f-424f-8d2e-2ce2a41fd98a" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719815 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b4b94b-944f-4382-9f7f-c8614d1b5de3" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719831 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abc9e45-7234-4330-962b-de93f1ce7a94" containerName="istio-proxy" Oct 30 19:38:04.720884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.719845 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8930e474-eb64-4d26-abe4-45d6f1d9c0df" containerName="maas-api" Oct 30 19:38:04.734369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.733755 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.754894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.752882 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzpp\" (UniqueName: \"kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp\") pod \"payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.754894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.752959 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.754894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.754065 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d651c4-dockercfg-b5jsz\"" Oct 30 19:38:04.792244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.791270 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d651c4\"" Oct 30 19:38:04.855481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.854841 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzpp\" (UniqueName: \"kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp\") pod \"payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.855786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.855734 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.858026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.856859 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.875071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.874324 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.875416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.875316 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9"] Oct 30 19:38:04.909291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.908615 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:04.909291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.909043 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9"] Oct 30 19:38:04.909291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.909198 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzpp\" (UniqueName: \"kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp\") pod \"payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:04.942841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:04.942744 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv"] Oct 30 19:38:05.070885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.059146 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.070885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.059320 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjrh\" (UniqueName: \"kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh\") pod \"payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.128703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.127347 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:05.177980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.177456 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjrh\" (UniqueName: \"kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh\") pod \"payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.177980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.177794 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.178491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.178473 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.273351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.273313 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjrh\" (UniqueName: \"kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh\") pod \"payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.323655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.323601 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:05.479101 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:05.479070 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c930635_2467_4bc3_94fb_80f8bf3361c8.slice/crio-c4bd8194c79418021982c47af1c959ec567b59805ce14f187effa693812b3b53 WatchSource:0}: Error finding container c4bd8194c79418021982c47af1c959ec567b59805ce14f187effa693812b3b53: Status 404 returned error can't find the container with id c4bd8194c79418021982c47af1c959ec567b59805ce14f187effa693812b3b53 Oct 30 19:38:05.480952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.480856 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv"] Oct 30 19:38:05.554968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.554734 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:05.624946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.617240 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:05.802291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.802222 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:05.931575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:05.931536 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" event={"ID":"8c930635-2467-4bc3-94fb-80f8bf3361c8","Type":"ContainerStarted","Data":"c4bd8194c79418021982c47af1c959ec567b59805ce14f187effa693812b3b53"} Oct 30 19:38:06.275458 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:06.275429 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed81f2a_e68a_42ab_b845_758d39b5fbae.slice/crio-987231154a8d048df8276b5a2facea513c5d082cc627abde0ba288fbc262caca WatchSource:0}: Error finding container 987231154a8d048df8276b5a2facea513c5d082cc627abde0ba288fbc262caca: Status 404 returned error can't find the container with id 987231154a8d048df8276b5a2facea513c5d082cc627abde0ba288fbc262caca Oct 30 19:38:06.277439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.277415 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9"] Oct 30 19:38:06.605955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.604111 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:06.801562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.801522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:06.941942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.941901 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" event={"ID":"4ed81f2a-e68a-42ab-b845-758d39b5fbae","Type":"ContainerStarted","Data":"616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3"} Oct 30 19:38:06.942136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.942000 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" event={"ID":"4ed81f2a-e68a-42ab-b845-758d39b5fbae","Type":"ContainerStarted","Data":"987231154a8d048df8276b5a2facea513c5d082cc627abde0ba288fbc262caca"} Oct 30 19:38:06.942214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.942146 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:06.943908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.943882 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" event={"ID":"8c930635-2467-4bc3-94fb-80f8bf3361c8","Type":"ContainerStarted","Data":"15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa"} Oct 30 19:38:06.944199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.944180 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:06.975850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:06.975777 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" podStartSLOduration=2.9757579229999997 podStartE2EDuration="2.975757923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:06.968496045 +0000 UTC m=+1256.926315235" watchObservedRunningTime="2026-10-30 19:38:06.975757923 +0000 UTC m=+1256.933577112" Oct 30 19:38:07.011323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:07.011266 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" podStartSLOduration=3.011248876 podStartE2EDuration="3.011248876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:07.010014986 +0000 UTC m=+1256.967834188" watchObservedRunningTime="2026-10-30 19:38:07.011248876 +0000 UTC m=+1256.969068078" Oct 30 19:38:07.736275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:07.736238 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:07.933713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:07.933625 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:08.801679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:08.801651 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:09.053745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:09.053657 6514 generic.go:356] "Generic (PLEG): container finished" podID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerID="61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6" exitCode=0 Oct 30 19:38:09.053929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:09.053727 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" event={"ID":"8ba39fa3-0b84-4a9f-98b2-faad69a3720f","Type":"ContainerDied","Data":"61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6"} Oct 30 19:38:09.054212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:09.054197 6514 scope.go:122] "RemoveContainer" containerID="61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6" Oct 30 19:38:09.056026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:09.055998 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:09.056154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:09.056085 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:09.056154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:09.056136 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:10.024015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:10.023979 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:10.153887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:10.148582 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" event={"ID":"8ba39fa3-0b84-4a9f-98b2-faad69a3720f","Type":"ContainerStarted","Data":"efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd"} Oct 30 19:38:10.851588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:10.851551 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:10.866944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:10.866918 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:38:10.868000 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:10.867977 6514 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:38:10.868068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:10.868037 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:38:11.316890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.308282 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:11.440755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.440707 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:11.636161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.636080 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp"] Oct 30 19:38:11.643421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.636330 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" containerID="cri-o://b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a" gracePeriod=30 Oct 30 19:38:11.646887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.646652 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw"] Oct 30 19:38:11.650887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.648117 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" containerID="cri-o://b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4" gracePeriod=30 Oct 30 19:38:11.669059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.669030 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:11.833956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.825332 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:11.876895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.876661 6514 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:38:11.876895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.876722 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:38:11.915946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.913269 6514 scope.go:122] "RemoveContainer" containerID="eb387434a26e500a04e9ffbc11b5737d77be559f84898e3d0a101d97af5e7a33" Oct 30 19:38:11.948033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:11.939249 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:12.736155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:12.736126 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:12.867564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:12.867528 6514 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:38:12.867725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:12.867580 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:38:13.665731 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:13.665699 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:13.867748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:13.867714 6514 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:38:13.868181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:13.867776 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:38:14.247749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:14.247707 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:38:14.419407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:14.419353 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:14.670271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:14.670189 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:14.867327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:14.867292 6514 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:38:14.867486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:14.867355 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 19:38:15.259503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:15.259473 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:15.872528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:15.872494 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:38:15.873496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:15.873471 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:38:16.125959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.124037 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:16.777569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.777536 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5"] Oct 30 19:38:16.782878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.782842 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.791152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.791101 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-156838-serving-cert\"" Oct 30 19:38:16.807013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.806017 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5"] Oct 30 19:38:16.822011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.821413 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.822011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.821451 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwd5\" (UniqueName: \"kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.822011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.821504 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.822011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.821548 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.923135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.923058 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.923281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.923136 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwd5\" (UniqueName: \"kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.923281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.923256 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.923364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.923349 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.924318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.924288 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.926762 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.926737 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.928562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.928538 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.932581 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.932542 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwd5\" (UniqueName: \"kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5\") pod \"maas-api-e2e-api-b-156838-74b98c57df-nl9w5\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:16.935479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.935453 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:16.984055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:16.983913 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:38:17.087214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.087105 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:17.130107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.130079 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:17.141715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.130771 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.141908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.141772 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.141908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.141821 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.142055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.141927 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.142055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.142037 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.142164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.142075 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.142164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.142129 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.142276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.142188 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.142352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.142300 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy\") pod \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\" (UID: \"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe\") " Oct 30 19:38:17.143578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.141024 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:17.143578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.143049 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.143929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.143764 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.144077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.144053 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.144165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.144114 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.147141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.145065 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.147340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.147324 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.147495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.147481 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.147657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.147639 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.147740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.147663 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.147740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.146550 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.150265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.150244 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "kube-api-access-jfbmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.152760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.152739 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:38:17.152859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.152734 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.157017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.156987 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo" pod "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" (UID: "aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:17.221468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.221439 6514 generic.go:356] "Generic (PLEG): container finished" podID="90471c40-e5e4-400d-9838-18151f77cd3c" containerID="b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4" exitCode=0 Oct 30 19:38:17.221686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.221665 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" event={"ID":"90471c40-e5e4-400d-9838-18151f77cd3c","Type":"ContainerDied","Data":"b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4"} Oct 30 19:38:17.221773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.221730 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" event={"ID":"90471c40-e5e4-400d-9838-18151f77cd3c","Type":"ContainerDied","Data":"a09e1661e020ebf21ca552d0906fad9f78dd2647e177f446151716e4bc91bfc6"} Oct 30 19:38:17.221832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.221769 6514 scope.go:122] "RemoveContainer" containerID="b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4" Oct 30 19:38:17.222030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.222010 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw" Oct 30 19:38:17.225663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.225633 6514 generic.go:356] "Generic (PLEG): container finished" podID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerID="b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a" exitCode=0 Oct 30 19:38:17.225840 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.225813 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" Oct 30 19:38:17.226004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.225978 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" event={"ID":"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe","Type":"ContainerDied","Data":"b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a"} Oct 30 19:38:17.226184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.226163 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp" event={"ID":"aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe","Type":"ContainerDied","Data":"4a1d669b529d3658430eea72d3aefcc3f122c1806819e6438ab17bf5f48d0449"} Oct 30 19:38:17.244051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.244031 6514 scope.go:122] "RemoveContainer" containerID="b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4" Oct 30 19:38:17.244395 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:17.244363 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4\": container with ID starting with b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4 not found: ID does not exist" containerID="b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4" Oct 30 19:38:17.244482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.244407 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4"} err="failed to get container status \"b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4\": rpc error: code = NotFound desc = could not find container \"b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4\": container with ID starting with b9e57bca6589571ae03fd09a42babdaf2df4a972088e369a60c46781c9cd5aa4 not found: ID does not exist" Oct 30 19:38:17.244482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.244433 6514 scope.go:122] "RemoveContainer" containerID="b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a" Oct 30 19:38:17.248673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.248647 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.248771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.248689 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.248771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.248721 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.248966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.248797 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.248966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.248825 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.248966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.248908 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.249138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.249048 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.249230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.249210 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.249291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.249255 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs\") pod \"90471c40-e5e4-400d-9838-18151f77cd3c\" (UID: \"90471c40-e5e4-400d-9838-18151f77cd3c\") " Oct 30 19:38:17.250178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.250155 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfbmm\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-kube-api-access-jfbmm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.250278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.250182 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.250278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.250196 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.250278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.250208 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.251171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.251145 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.253884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.253824 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.254396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.254368 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:17.257079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.257049 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.261451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.261424 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.262575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.262552 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.263803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.263780 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.264623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.264602 6514 scope.go:122] "RemoveContainer" containerID="b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a" Oct 30 19:38:17.265144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.264845 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:17.265434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.265407 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg" pod "90471c40-e5e4-400d-9838-18151f77cd3c" (UID: "90471c40-e5e4-400d-9838-18151f77cd3c"). InnerVolumeSpecName "kube-api-access-g24pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.265580 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:17.265514 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a\": container with ID starting with b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a not found: ID does not exist" containerID="b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a" Oct 30 19:38:17.265580 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.265557 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a"} err="failed to get container status \"b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a\": rpc error: code = NotFound desc = could not find container \"b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a\": container with ID starting with b5a12adab71b70ef2749ac4852d3140f5f702a8da763f8d60815dab061b0dc9a not found: ID does not exist" Oct 30 19:38:17.289698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.289668 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp"] Oct 30 19:38:17.331957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.311401 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jrrrp"] Oct 30 19:38:17.361626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.361597 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90471c40-e5e4-400d-9838-18151f77cd3c-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.363770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.363741 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.363965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.363949 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.364071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.364058 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.364158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.364146 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.364262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.364251 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.364347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.364336 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90471c40-e5e4-400d-9838-18151f77cd3c-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.364428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.364416 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g24pg\" (UniqueName: \"kubernetes.io/projected/90471c40-e5e4-400d-9838-18151f77cd3c-kube-api-access-g24pg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.364507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.364496 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90471c40-e5e4-400d-9838-18151f77cd3c-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.478921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.478889 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5"] Oct 30 19:38:17.479289 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:17.479259 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb64de1_b9b1_49b4_bc78_8cbc6d1b3245.slice/crio-e4eb6ac059ce2f5ac14a321dfed678ecba26695b9917f22d5f9cc44177e048b5 WatchSource:0}: Error finding container e4eb6ac059ce2f5ac14a321dfed678ecba26695b9917f22d5f9cc44177e048b5: Status 404 returned error can't find the container with id e4eb6ac059ce2f5ac14a321dfed678ecba26695b9917f22d5f9cc44177e048b5 Oct 30 19:38:17.662094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.661957 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw"] Oct 30 19:38:17.667419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.667383 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-ncnjw"] Oct 30 19:38:17.932643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:17.932605 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:18.040258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.040207 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:38:18.040258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.041967 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:38:18.164538 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.164510 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" path="/var/lib/kubelet/pods/90471c40-e5e4-400d-9838-18151f77cd3c/volumes" Oct 30 19:38:18.165190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.165169 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" path="/var/lib/kubelet/pods/aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe/volumes" Oct 30 19:38:18.238476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.238389 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" event={"ID":"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245","Type":"ContainerStarted","Data":"f8cf93e5c7622ddb0a0aea418341c44a5deff495c1fe3b8aa7bb0e2f2e641d74"} Oct 30 19:38:18.238476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.238448 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" event={"ID":"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245","Type":"ContainerStarted","Data":"e4eb6ac059ce2f5ac14a321dfed678ecba26695b9917f22d5f9cc44177e048b5"} Oct 30 19:38:18.238681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.238478 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:18.278015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.277962 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" podStartSLOduration=1.7369763599999999 podStartE2EDuration="2.277949609s" totalImagesPullingTime="540.973249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:18.277433543 +0000 UTC m=+1268.235252735" watchObservedRunningTime="2026-10-30 19:38:18.277949609 +0000 UTC m=+1268.235768798" Oct 30 19:38:18.904931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:18.904898 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:19.712306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:19.712273 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:19.755417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:19.755385 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz"] Oct 30 19:38:19.840702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:19.840671 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:19.967652 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:19.967550 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:20.702208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:20.702166 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:20.883495 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:20.883460 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:38:21.093407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:21.093321 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:21.564423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:21.564384 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.180674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.180583 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd"] Oct 30 19:38:22.181878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.181837 6514 cpu_manager.go:414] "containerMap: removing container" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" Oct 30 19:38:22.182013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.181886 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" Oct 30 19:38:22.182013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.181946 6514 cpu_manager.go:414] "containerMap: removing container" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" Oct 30 19:38:22.182013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.181956 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" Oct 30 19:38:22.182203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.182123 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="90471c40-e5e4-400d-9838-18151f77cd3c" containerName="istio-proxy" Oct 30 19:38:22.182203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.182147 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaafa3a2-1a01-4a5f-ba25-e1e48a5e4dbe" containerName="istio-proxy" Oct 30 19:38:22.188138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.188112 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.198314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.198282 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd"] Oct 30 19:38:22.199792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.199761 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d651c4-serving-cert\"" Oct 30 19:38:22.215131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.215102 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.215353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.215329 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4tk\" (UniqueName: \"kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.215700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.215676 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.215785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.215729 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.246646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.246595 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.322031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.321989 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.324647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.324620 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4tk\" (UniqueName: \"kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.326198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.326170 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.327918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.327896 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.328393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.323143 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.330348 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:22.328128 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-serving-cert: secret "maas-api-e2e-inf-b-d651c4-serving-cert" not found Oct 30 19:38:22.330538 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:22.330526 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls podName:32e143cc-e88c-417b-9dde-8e6b9310f58d nodeName:}" failed. No retries permitted until 2026-10-30 19:38:22.830502458 +0000 UTC m=+1272.788321641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls") pod "maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" (UID: "32e143cc-e88c-417b-9dde-8e6b9310f58d") : secret "maas-api-e2e-inf-b-d651c4-serving-cert" not found Oct 30 19:38:22.331504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.331485 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.348836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.348189 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4tk\" (UniqueName: \"kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.380638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.380050 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.507350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.507256 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.839750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.839135 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:22.859919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:22.858384 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:23.112954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:23.112854 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:23.154011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:23.153982 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:23.438270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:23.438242 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd"] Oct 30 19:38:23.439455 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:23.439428 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e143cc_e88c_417b_9dde_8e6b9310f58d.slice/crio-47848dd01160044f86bdbd00783efe7ac36b55ae7cfe1308ee655f5f9481c130 WatchSource:0}: Error finding container 47848dd01160044f86bdbd00783efe7ac36b55ae7cfe1308ee655f5f9481c130: Status 404 returned error can't find the container with id 47848dd01160044f86bdbd00783efe7ac36b55ae7cfe1308ee655f5f9481c130 Oct 30 19:38:23.887197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:23.887171 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:24.255298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:24.255267 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:38:24.348571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:24.348534 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" event={"ID":"32e143cc-e88c-417b-9dde-8e6b9310f58d","Type":"ContainerStarted","Data":"6f844afdc3bbd1884977b10823dbedae98a504dddd560095d53b1d507fb3de10"} Oct 30 19:38:24.348735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:24.348592 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" event={"ID":"32e143cc-e88c-417b-9dde-8e6b9310f58d","Type":"ContainerStarted","Data":"47848dd01160044f86bdbd00783efe7ac36b55ae7cfe1308ee655f5f9481c130"} Oct 30 19:38:24.348735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:24.348658 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:24.369301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:24.369256 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" podStartSLOduration=1.9662134610000002 podStartE2EDuration="2.369243637s" totalImagesPullingTime="403.030176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:24.367719569 +0000 UTC m=+1274.325538763" watchObservedRunningTime="2026-10-30 19:38:24.369243637 +0000 UTC m=+1274.327062826" Oct 30 19:38:24.756888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:24.756123 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" podUID="dc005260-0822-4c07-a4b0-caae7db8468c" containerName="payload-processing" containerID="cri-o://45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e" gracePeriod=25 Oct 30 19:38:25.004304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.004059 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:25.132740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.105352 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:38:25.271847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.271816 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k\" (UniqueName: \"kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k\") pod \"dc005260-0822-4c07-a4b0-caae7db8468c\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " Oct 30 19:38:25.272221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.272198 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume\") pod \"dc005260-0822-4c07-a4b0-caae7db8468c\" (UID: \"dc005260-0822-4c07-a4b0-caae7db8468c\") " Oct 30 19:38:25.272543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.272516 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume" pod "dc005260-0822-4c07-a4b0-caae7db8468c" (UID: "dc005260-0822-4c07-a4b0-caae7db8468c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:25.273077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.273050 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc005260-0822-4c07-a4b0-caae7db8468c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:25.274073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.274054 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k" pod "dc005260-0822-4c07-a4b0-caae7db8468c" (UID: "dc005260-0822-4c07-a4b0-caae7db8468c"). InnerVolumeSpecName "kube-api-access-7sd2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:25.357958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.357928 6514 generic.go:356] "Generic (PLEG): container finished" podID="dc005260-0822-4c07-a4b0-caae7db8468c" containerID="45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e" exitCode=0 Oct 30 19:38:25.358140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.357975 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" Oct 30 19:38:25.358140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.358006 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" event={"ID":"dc005260-0822-4c07-a4b0-caae7db8468c","Type":"ContainerDied","Data":"45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e"} Oct 30 19:38:25.358140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.358070 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz" event={"ID":"dc005260-0822-4c07-a4b0-caae7db8468c","Type":"ContainerDied","Data":"633eb9e17a5cfb32857fbc7b44946ed52f8858007e3b1018b75cca044c93b75b"} Oct 30 19:38:25.358140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.358101 6514 scope.go:122] "RemoveContainer" containerID="45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e" Oct 30 19:38:25.369433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.369414 6514 scope.go:122] "RemoveContainer" containerID="45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e" Oct 30 19:38:25.369692 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:25.369671 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e\": container with ID starting with 45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e not found: ID does not exist" containerID="45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e" Oct 30 19:38:25.369759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.369697 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e"} err="failed to get container status \"45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e\": rpc error: code = NotFound desc = could not find container \"45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e\": container with ID starting with 45666cd473b94ebbb067857d5c6179d2d74bff4ed2b8618f81c6fd6ac1b40d2e not found: ID does not exist" Oct 30 19:38:25.378036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.378009 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sd2k\" (UniqueName: \"kubernetes.io/projected/dc005260-0822-4c07-a4b0-caae7db8468c-kube-api-access-7sd2k\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:25.391124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.391099 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz"] Oct 30 19:38:25.395421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.395397 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-395ccc-656669bc5b-sg8dz"] Oct 30 19:38:25.837690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.837660 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:25.963254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:25.963224 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.161457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:26.161386 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc005260-0822-4c07-a4b0-caae7db8468c" path="/var/lib/kubelet/pods/dc005260-0822-4c07-a4b0-caae7db8468c/volumes" Oct 30 19:38:26.432421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:26.432391 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.772226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:26.772148 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.891236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:26.891199 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:27.038240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:27.034882 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:27.561478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:27.561437 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp"] Oct 30 19:38:27.561723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:27.561640 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" podUID="52e5bfbd-00da-4e97-96d5-ab8a2683664e" containerName="maas-api" containerID="cri-o://7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a" gracePeriod=30 Oct 30 19:38:27.629075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:27.629046 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:27.817505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:27.817301 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:28.027595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.027430 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:28.220489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.220461 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:38:28.282226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.282194 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:28.301696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.301670 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca\") pod \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " Oct 30 19:38:28.301859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.301731 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls\") pod \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " Oct 30 19:38:28.301859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.301798 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g\" (UniqueName: \"kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g\") pod \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " Oct 30 19:38:28.301859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.301874 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs\") pod \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\" (UID: \"52e5bfbd-00da-4e97-96d5-ab8a2683664e\") " Oct 30 19:38:28.302107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.302028 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca" pod "52e5bfbd-00da-4e97-96d5-ab8a2683664e" (UID: "52e5bfbd-00da-4e97-96d5-ab8a2683664e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:28.302494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.302471 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5bfbd-00da-4e97-96d5-ab8a2683664e-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:28.303904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.303874 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs" pod "52e5bfbd-00da-4e97-96d5-ab8a2683664e" (UID: "52e5bfbd-00da-4e97-96d5-ab8a2683664e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:28.304005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.303904 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls" pod "52e5bfbd-00da-4e97-96d5-ab8a2683664e" (UID: "52e5bfbd-00da-4e97-96d5-ab8a2683664e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:28.304005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.303908 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g" pod "52e5bfbd-00da-4e97-96d5-ab8a2683664e" (UID: "52e5bfbd-00da-4e97-96d5-ab8a2683664e"). InnerVolumeSpecName "kube-api-access-wvv2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:28.408575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.404816 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:28.408575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.404855 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvv2g\" (UniqueName: \"kubernetes.io/projected/52e5bfbd-00da-4e97-96d5-ab8a2683664e-kube-api-access-wvv2g\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:28.408575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.404888 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5bfbd-00da-4e97-96d5-ab8a2683664e-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:28.451961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.450938 6514 generic.go:356] "Generic (PLEG): container finished" podID="52e5bfbd-00da-4e97-96d5-ab8a2683664e" containerID="7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a" exitCode=0 Oct 30 19:38:28.451961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.451003 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" Oct 30 19:38:28.451961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.451012 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" event={"ID":"52e5bfbd-00da-4e97-96d5-ab8a2683664e","Type":"ContainerDied","Data":"7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a"} Oct 30 19:38:28.451961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.451080 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp" event={"ID":"52e5bfbd-00da-4e97-96d5-ab8a2683664e","Type":"ContainerDied","Data":"bfce4015f57f4815c973e2fedec766b97ea2a103d9ae7a1b9fcf839a4795dc0d"} Oct 30 19:38:28.451961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.451107 6514 scope.go:122] "RemoveContainer" containerID="7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a" Oct 30 19:38:28.463689 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.463668 6514 scope.go:122] "RemoveContainer" containerID="7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a" Oct 30 19:38:28.464010 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:28.463984 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a\": container with ID starting with 7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a not found: ID does not exist" containerID="7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a" Oct 30 19:38:28.464069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.464019 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a"} err="failed to get container status \"7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a\": rpc error: code = NotFound desc = could not find container \"7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a\": container with ID starting with 7e2b8b3c4cca5ee3dede75518e06414bdd4bf8afda0afd70916787fd250a682a not found: ID does not exist" Oct 30 19:38:28.476550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.476524 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp"] Oct 30 19:38:28.606451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.606420 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:28.619681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:28.619618 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-395ccc-787bc65c9d-x7zkp"] Oct 30 19:38:29.127841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:29.127810 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:29.882310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:29.882267 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:30.161141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:30.161062 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e5bfbd-00da-4e97-96d5-ab8a2683664e" path="/var/lib/kubelet/pods/52e5bfbd-00da-4e97-96d5-ab8a2683664e/volumes" Oct 30 19:38:30.364486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:30.364456 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:38:30.790596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:30.790556 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:31.065076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:31.064982 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:31.276791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:31.276755 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:38:32.029672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.029639 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:32.092368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.092334 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d"] Oct 30 19:38:32.093170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.093154 6514 cpu_manager.go:414] "containerMap: removing container" podUID="dc005260-0822-4c07-a4b0-caae7db8468c" containerName="payload-processing" Oct 30 19:38:32.093224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.093171 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc005260-0822-4c07-a4b0-caae7db8468c" containerName="payload-processing" Oct 30 19:38:32.093224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.093211 6514 cpu_manager.go:414] "containerMap: removing container" podUID="52e5bfbd-00da-4e97-96d5-ab8a2683664e" containerName="maas-api" Oct 30 19:38:32.093224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.093217 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e5bfbd-00da-4e97-96d5-ab8a2683664e" containerName="maas-api" Oct 30 19:38:32.093371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.093327 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc005260-0822-4c07-a4b0-caae7db8468c" containerName="payload-processing" Oct 30 19:38:32.093371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.093345 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e5bfbd-00da-4e97-96d5-ab8a2683664e" containerName="maas-api" Oct 30 19:38:32.097515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.097487 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.107088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.107060 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d"] Oct 30 19:38:32.150844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.150809 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:32.172992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.172953 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1e13df\"" Oct 30 19:38:32.173174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.173096 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1e13df-dockercfg-bgd85\"" Oct 30 19:38:32.279212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.279173 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8jx\" (UniqueName: \"kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx\") pod \"payload-processing-e2e-inf-b-1e13df-68d544b494-c828d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.280735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.280668 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1e13df-68d544b494-c828d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.384188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.384150 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8jx\" (UniqueName: \"kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx\") pod \"payload-processing-e2e-inf-b-1e13df-68d544b494-c828d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.384948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.384918 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1e13df-68d544b494-c828d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.386553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.386531 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1e13df-68d544b494-c828d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.412630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.412600 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8jx\" (UniqueName: \"kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx\") pod \"payload-processing-e2e-inf-b-1e13df-68d544b494-c828d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.420518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.420487 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:32.862158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:32.862065 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d"] Oct 30 19:38:32.924201 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:32.920198 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3531267e_df22_4f96_90a8_11244e83f68d.slice/crio-df1b2cfdd624c4e51d8bebf6921dcb9e921bd97074b06233b1ffeff8d0f58fad WatchSource:0}: Error finding container df1b2cfdd624c4e51d8bebf6921dcb9e921bd97074b06233b1ffeff8d0f58fad: Status 404 returned error can't find the container with id df1b2cfdd624c4e51d8bebf6921dcb9e921bd97074b06233b1ffeff8d0f58fad Oct 30 19:38:33.322680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:33.322639 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:33.528292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:33.528257 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" event={"ID":"3531267e-df22-4f96-90a8-11244e83f68d","Type":"ContainerStarted","Data":"4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d"} Oct 30 19:38:33.528498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:33.528318 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" event={"ID":"3531267e-df22-4f96-90a8-11244e83f68d","Type":"ContainerStarted","Data":"df1b2cfdd624c4e51d8bebf6921dcb9e921bd97074b06233b1ffeff8d0f58fad"} Oct 30 19:38:33.528498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:33.528369 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:33.556632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:33.556584 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" podStartSLOduration=1.5565702799999999 podStartE2EDuration="1.55657028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:33.554072002 +0000 UTC m=+1283.511891194" watchObservedRunningTime="2026-10-30 19:38:33.55657028 +0000 UTC m=+1283.514389468" Oct 30 19:38:34.055534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:34.055498 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:34.957396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:34.957349 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:36.175974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:36.175931 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:36.722838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:36.722802 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:36.881107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:36.881075 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:37.043797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:37.043694 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:37.295300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:37.295202 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:38.198212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:38.198179 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:39.032330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:39.032299 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:39.585123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:39.585089 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:39.828382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:39.828333 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:40.762736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:40.762706 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:40.959432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:40.959368 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:41.603327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:41.603297 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:41.916446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:41.916369 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.145094 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.145060 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx"] Oct 30 19:38:42.151404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.151375 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.169962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.169904 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-22af66-openshift-default-dockercfg-6d5f5\"" Oct 30 19:38:42.172836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.172815 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx"] Oct 30 19:38:42.224640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.224606 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.305035 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.304985 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgc84\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.305318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.305295 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.305472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.305455 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.305669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.305639 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.305826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.305808 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.322895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.306331 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.323196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.323168 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.323330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.323314 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.323467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.323452 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.426372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.426284 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.426613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.426592 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.426720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.426708 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.426812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.426801 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.426960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.426946 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgc84\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.427095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.427081 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.427220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.427208 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.427352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.427341 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.427441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.427428 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.428364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.428338 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.447258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.433278 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.447449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.433850 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.447546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.434150 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.448482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.448447 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.448740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.448688 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.473261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.472656 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.473814 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.473785 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.485893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.483998 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgc84\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.492901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.489519 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token\") pod \"e2e-api-b-22af66-openshift-default-5f7c494769-gqftx\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.768075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.768041 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:42.927110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:42.927072 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:43.085935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.082769 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:43.128961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.107810 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx"] Oct 30 19:38:43.146891 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:43.141202 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa196ee8_c199_4129_8870_2ab2870a75fd.slice/crio-a2911fefebf5cc869424ff6e24a36779bd01736bb8c7b07213d6b06afa196b15 WatchSource:0}: Error finding container a2911fefebf5cc869424ff6e24a36779bd01736bb8c7b07213d6b06afa196b15: Status 404 returned error can't find the container with id a2911fefebf5cc869424ff6e24a36779bd01736bb8c7b07213d6b06afa196b15 Oct 30 19:38:43.161896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.155986 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:43.161896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.156085 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:43.161896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.156131 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:43.243970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.243887 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:43.373555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.373521 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:43.637271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.637190 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:43.670113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.670079 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" event={"ID":"aa196ee8-c199-4129-8870-2ab2870a75fd","Type":"ContainerStarted","Data":"be96eb88197a1d4ee1372563fda37ff8b794d5bd5d78e24bb21a1040ccdc51fc"} Oct 30 19:38:43.670292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.670209 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" event={"ID":"aa196ee8-c199-4129-8870-2ab2870a75fd","Type":"ContainerStarted","Data":"a2911fefebf5cc869424ff6e24a36779bd01736bb8c7b07213d6b06afa196b15"} Oct 30 19:38:43.699207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.699152 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podStartSLOduration=1.69913729 podStartE2EDuration="1.69913729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:43.697731192 +0000 UTC m=+1293.655550381" watchObservedRunningTime="2026-10-30 19:38:43.69913729 +0000 UTC m=+1293.656956479" Oct 30 19:38:43.768441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:43.768407 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:44.338245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.338208 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl"] Oct 30 19:38:44.344635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.344609 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.354625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.354601 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-89a3f6\"/\"default-dockercfg-c82ds\"" Oct 30 19:38:44.355984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.355959 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-89a3f6\"/\"test-model-89a3f6-kserve-self-signed-certs\"" Oct 30 19:38:44.358342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.358307 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-89a3f6\"/\"openshift-service-ca.crt\"" Oct 30 19:38:44.358342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.358326 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-89a3f6\"/\"kube-root-ca.crt\"" Oct 30 19:38:44.368132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.368111 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl"] Oct 30 19:38:44.415858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.415830 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.416050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.415948 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.416131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.416070 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.416212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.416195 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.416320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.416303 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqw4\" (UniqueName: \"kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.416396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.416378 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.517727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.517690 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.517977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.517773 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.517977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.517838 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.517977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.517928 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqw4\" (UniqueName: \"kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.518235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.518046 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.518235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.518099 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.518375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.518253 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.518527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.518504 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.518622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.518608 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.520256 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.520223 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.520525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.520509 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.533280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.533255 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqw4\" (UniqueName: \"kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4\") pod \"test-model-89a3f6-kserve-6456cdf69c-dk7xl\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.539130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.539111 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:38:44.553230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.553200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:44.665933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.665813 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:44.826886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.820854 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:44.961304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:44.961275 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:45.154822 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:45.154787 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52d0848f_74ed_4310_b6cb_53b7676793e5.slice/crio-5e89f9bc35e7f22266a75745eb434c1dfcc0836a63ad2e119b73996d16e701ea WatchSource:0}: Error finding container 5e89f9bc35e7f22266a75745eb434c1dfcc0836a63ad2e119b73996d16e701ea: Status 404 returned error can't find the container with id 5e89f9bc35e7f22266a75745eb434c1dfcc0836a63ad2e119b73996d16e701ea Oct 30 19:38:45.155182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.155068 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl"] Oct 30 19:38:45.689851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.689813 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" event={"ID":"52d0848f-74ed-4310-b6cb-53b7676793e5","Type":"ContainerStarted","Data":"0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0"} Oct 30 19:38:45.690356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.689876 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" event={"ID":"52d0848f-74ed-4310-b6cb-53b7676793e5","Type":"ContainerStarted","Data":"5e89f9bc35e7f22266a75745eb434c1dfcc0836a63ad2e119b73996d16e701ea"} Oct 30 19:38:45.690441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.690383 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:45.691549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.691526 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:38:45.712304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.712262 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" podStartSLOduration=1.7122473710000001 podStartE2EDuration="1.712247371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:45.711889172 +0000 UTC m=+1295.669708353" watchObservedRunningTime="2026-10-30 19:38:45.712247371 +0000 UTC m=+1295.670066559" Oct 30 19:38:45.755208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:45.755176 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:46.162566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:46.162478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:46.304789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:46.304756 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:46.688888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:46.677068 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:46.715884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:46.715074 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:47.502068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:47.502036 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:47.762940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:47.750500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:47.976911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:47.975533 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:48.097772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:48.097676 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:48.742738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:48.742704 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:48.879995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:48.879962 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:49.931037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:49.931001 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:50.904987 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:50.904948 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:50.969381 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:38:50.969344 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:38:50.997587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:50.997556 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj"] Oct 30 19:38:51.096579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:51.096535 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:51.211283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:51.211251 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:51.415369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:51.415335 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:51.549168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:51.549068 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:51.776791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:51.776760 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:51.915811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:51.915690 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:52.561678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:52.561632 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:52.721337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:52.721257 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:53.017353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:53.017268 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:53.443452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:53.443403 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:54.362341 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:54.362295 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:54.666353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:54.666268 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:54.777599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:54.777554 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:55.453911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:55.445267 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:38:55.588007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:55.587966 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:55.999032 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:55.998949 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" podUID="9b528a4e-8af9-49d2-9ebf-da6359f9b06b" containerName="payload-processing" containerID="cri-o://cb230dd6badbdba1e42d37339680e99f6da111e51d7052e7310c4a02c7909456" gracePeriod=25 Oct 30 19:38:56.252323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.252235 6514 generic.go:356] "Generic (PLEG): container finished" podID="9b528a4e-8af9-49d2-9ebf-da6359f9b06b" containerID="cb230dd6badbdba1e42d37339680e99f6da111e51d7052e7310c4a02c7909456" exitCode=0 Oct 30 19:38:56.252323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.252308 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" event={"ID":"9b528a4e-8af9-49d2-9ebf-da6359f9b06b","Type":"ContainerDied","Data":"cb230dd6badbdba1e42d37339680e99f6da111e51d7052e7310c4a02c7909456"} Oct 30 19:38:56.555923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.555804 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:38:56.565326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.565290 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr\" (UniqueName: \"kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr\") pod \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " Oct 30 19:38:56.565462 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.565359 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume\") pod \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\" (UID: \"9b528a4e-8af9-49d2-9ebf-da6359f9b06b\") " Oct 30 19:38:56.566502 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.566476 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume" pod "9b528a4e-8af9-49d2-9ebf-da6359f9b06b" (UID: "9b528a4e-8af9-49d2-9ebf-da6359f9b06b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:56.568228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.568207 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr" pod "9b528a4e-8af9-49d2-9ebf-da6359f9b06b" (UID: "9b528a4e-8af9-49d2-9ebf-da6359f9b06b"). InnerVolumeSpecName "kube-api-access-pbtsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:56.650774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.650730 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:56.668298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.668263 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtsr\" (UniqueName: \"kubernetes.io/projected/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-kube-api-access-pbtsr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:56.668298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.668295 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b528a4e-8af9-49d2-9ebf-da6359f9b06b-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:56.790011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.789974 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:56.954294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:56.954243 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:57.604888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.599556 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" event={"ID":"9b528a4e-8af9-49d2-9ebf-da6359f9b06b","Type":"ContainerDied","Data":"3a7608aac584c64b9e2c405b849c6edabe726260423dbce0a55b6866b08ce6e0"} Oct 30 19:38:57.604888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.599638 6514 scope.go:122] "RemoveContainer" containerID="cb230dd6badbdba1e42d37339680e99f6da111e51d7052e7310c4a02c7909456" Oct 30 19:38:57.604888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.599840 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj" Oct 30 19:38:57.661887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.655786 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:57.709808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.709777 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj"] Oct 30 19:38:57.769436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.769410 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-82012b-64d6c8f676-wv8kj"] Oct 30 19:38:57.783575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.783549 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh"] Oct 30 19:38:57.786439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.786329 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9b528a4e-8af9-49d2-9ebf-da6359f9b06b" containerName="payload-processing" Oct 30 19:38:57.786601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.786585 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b528a4e-8af9-49d2-9ebf-da6359f9b06b" containerName="payload-processing" Oct 30 19:38:57.786967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.786949 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b528a4e-8af9-49d2-9ebf-da6359f9b06b" containerName="payload-processing" Oct 30 19:38:57.848326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.848288 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:57.848935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.848912 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh"] Oct 30 19:38:57.849257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.849201 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:57.860943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.858333 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d651c4\"/\"test-model-d651c4-kserve-self-signed-certs\"" Oct 30 19:38:57.860943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.858462 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d651c4\"/\"kube-root-ca.crt\"" Oct 30 19:38:57.860943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.858486 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d651c4\"/\"openshift-service-ca.crt\"" Oct 30 19:38:57.860943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.858586 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d651c4\"/\"default-dockercfg-6f785\"" Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.975217 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.976853 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.976910 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.977015 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.977104 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flm7r\" (UniqueName: \"kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.977192 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:57.982458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:57.977270 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.090888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.087493 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.090888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.087606 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.090888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.087792 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.090888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.087828 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.090888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.087951 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.090888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.088037 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flm7r\" (UniqueName: \"kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.107162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.105521 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.107358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.107189 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.138933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.138695 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.291891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.251552 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flm7r\" (UniqueName: \"kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.410942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.405494 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.410942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.410692 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:58.416886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.414144 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b528a4e-8af9-49d2-9ebf-da6359f9b06b" path="/var/lib/kubelet/pods/9b528a4e-8af9-49d2-9ebf-da6359f9b06b/volumes" Oct 30 19:38:58.417292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.417268 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm\") pod \"test-model-d651c4-kserve-8d886d889-wjxlh\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.480826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.480795 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:38:58.857007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:58.856977 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh"] Oct 30 19:38:58.858470 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:38:58.858439 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod686d6e36_5155_41cb_9d3d_cb96848abf79.slice/crio-b9147a6283564c98a9376c85ce1afe256471b017a0d3229afad44dd7ae093323 WatchSource:0}: Error finding container b9147a6283564c98a9376c85ce1afe256471b017a0d3229afad44dd7ae093323: Status 404 returned error can't find the container with id b9147a6283564c98a9376c85ce1afe256471b017a0d3229afad44dd7ae093323 Oct 30 19:38:59.125472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:59.125355 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.322888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:59.317447 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.682955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:59.672189 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.833759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:59.833722 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" event={"ID":"686d6e36-5155-41cb-9d3d-cb96848abf79","Type":"ContainerStarted","Data":"447c0111dff0a6f5faaf72bd4f25a4833131fe097f5e675912007c1c245dd02e"} Oct 30 19:38:59.833759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:59.833766 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" event={"ID":"686d6e36-5155-41cb-9d3d-cb96848abf79","Type":"ContainerStarted","Data":"b9147a6283564c98a9376c85ce1afe256471b017a0d3229afad44dd7ae093323"} Oct 30 19:38:59.858389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:38:59.858340 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" podStartSLOduration=2.858323096 podStartE2EDuration="2.858323096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:59.854918037 +0000 UTC m=+1309.812737228" watchObservedRunningTime="2026-10-30 19:38:59.858323096 +0000 UTC m=+1309.816142286" Oct 30 19:39:00.481585 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:00.481550 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:39:00.654404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:00.654274 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:01.474067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:01.473995 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:02.480540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:02.480457 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.246794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:03.246752 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.111362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.111244 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.744528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.744492 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.884576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.884543 6514 generic.go:356] "Generic (PLEG): container finished" podID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerID="85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79" exitCode=0 Oct 30 19:39:04.884765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.884618 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" event={"ID":"ab279feb-f6f1-4ea9-912a-8eef55566912","Type":"ContainerDied","Data":"85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79"} Oct 30 19:39:04.885215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.885195 6514 scope.go:122] "RemoveContainer" containerID="85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79" Oct 30 19:39:04.887616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.887580 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:04.887710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.887691 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:04.887761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:04.887742 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:05.469070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.469035 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:05.675246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.675170 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.901491 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" event={"ID":"ab279feb-f6f1-4ea9-912a-8eef55566912","Type":"ContainerStarted","Data":"69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2"} Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.904811 6514 generic.go:356] "Generic (PLEG): container finished" podID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerID="9ef9b1224be0ac04cb6c298089bd3fd1b3fb430e3653b53f41f0c71ed16be8c9" exitCode=0 Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.905022 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" event={"ID":"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7","Type":"ContainerDied","Data":"9ef9b1224be0ac04cb6c298089bd3fd1b3fb430e3653b53f41f0c71ed16be8c9"} Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.905498 6514 scope.go:122] "RemoveContainer" containerID="9ef9b1224be0ac04cb6c298089bd3fd1b3fb430e3653b53f41f0c71ed16be8c9" Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.907712 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.907796 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:05.912990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:05.907842 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:06.053372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.049426 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:39:06.053372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.051196 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:39:06.053372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.051258 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:39:06.053372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.053240 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:06.364889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.357466 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:06.529434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.529393 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:07.000890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.998000 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" event={"ID":"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7","Type":"ContainerStarted","Data":"822cae51d84eca02fdad2987056848326db1bf8a197240d0aad3297153c2703c"} Oct 30 19:39:07.000890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:06.999740 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:07.057441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:07.057401 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:39:07.057789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:07.057688 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:39:07.136906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:07.135480 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:07.787352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:07.787311 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:39:07.787754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:07.787458 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:39:07.787754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:07.787499 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:39:08.061093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.053202 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:39:08.061093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.053258 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:39:08.485984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.485947 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:39:08.543431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.543394 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:08.677886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.677839 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:39:08.786383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.786304 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:39:08.786383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:08.786354 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:39:09.050048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:09.049956 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:39:09.050048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:09.050003 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:39:09.078312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:09.078284 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:39:09.537888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:09.536625 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:09.786334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:09.786294 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:39:09.786518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:09.786343 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:39:10.050166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:10.050127 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:39:10.050557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:10.050187 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:39:10.578136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:10.578104 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:10.790416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:10.790382 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:39:10.818047 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:10.818006 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:39:11.049649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:11.049613 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:39:11.049841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:11.049665 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:39:11.786422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:11.786362 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:39:11.787060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:11.786436 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:39:12.022466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.022427 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:12.162992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.162910 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:12.228890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.213747 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:39:12.324844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.324789 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:12.328697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.328668 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:39:12.786261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.786223 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:39:12.786441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:12.786281 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:39:13.159188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:13.159104 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:39:13.292164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:13.292134 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:13.791160 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:13.791131 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:39:13.791611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:13.791526 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:39:13.792131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:13.792112 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:39:14.055916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:14.055843 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:14.780628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:14.780593 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:15.661475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:15.661444 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:15.787557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:15.787520 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:16.613904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:16.604589 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:16.870938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:16.858719 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:17.709318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:17.709282 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:17.939889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:17.931423 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:18.083331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:18.083247 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:18.729165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:18.729135 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:19.256760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:19.256728 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:20.897732 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:39:20.897688 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:39:21.618140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:21.618107 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.725555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:21.725522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.843386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:21.843345 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:22.770940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:22.770907 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:23.494390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:23.494357 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:24.290990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:24.290953 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:25.370908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:25.370462 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:26.643785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:26.643754 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:26.847199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:26.847164 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:27.629250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:27.629215 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:28.691978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:28.691938 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:29.254914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.254880 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65"] Oct 30 19:39:29.255385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.255358 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" podUID="393374a1-9e7d-4867-a325-7dfd58126370" containerName="maas-api" containerID="cri-o://79bd292c98af0c9836bb415bb4d415de97597a28b410ebcade083415e743c080" gracePeriod=30 Oct 30 19:39:29.320090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.320058 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:29.565427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.551659 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:29.800180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.800134 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:29.808409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.808343 6514 generic.go:356] "Generic (PLEG): container finished" podID="393374a1-9e7d-4867-a325-7dfd58126370" containerID="79bd292c98af0c9836bb415bb4d415de97597a28b410ebcade083415e743c080" exitCode=0 Oct 30 19:39:29.809763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.809735 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" event={"ID":"393374a1-9e7d-4867-a325-7dfd58126370","Type":"ContainerDied","Data":"79bd292c98af0c9836bb415bb4d415de97597a28b410ebcade083415e743c080"} Oct 30 19:39:29.894926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.894661 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:39:29.974043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:29.974006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:30.030824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.030791 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls\") pod \"393374a1-9e7d-4867-a325-7dfd58126370\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " Oct 30 19:39:30.031037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.030835 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca\") pod \"393374a1-9e7d-4867-a325-7dfd58126370\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " Oct 30 19:39:30.031037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.030973 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj\" (UniqueName: \"kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj\") pod \"393374a1-9e7d-4867-a325-7dfd58126370\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " Oct 30 19:39:30.031184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.031101 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs\") pod \"393374a1-9e7d-4867-a325-7dfd58126370\" (UID: \"393374a1-9e7d-4867-a325-7dfd58126370\") " Oct 30 19:39:30.031246 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.031207 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca" pod "393374a1-9e7d-4867-a325-7dfd58126370" (UID: "393374a1-9e7d-4867-a325-7dfd58126370"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:30.031794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.031765 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393374a1-9e7d-4867-a325-7dfd58126370-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:30.033313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.033282 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj" pod "393374a1-9e7d-4867-a325-7dfd58126370" (UID: "393374a1-9e7d-4867-a325-7dfd58126370"). InnerVolumeSpecName "kube-api-access-6znqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:30.033411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.033373 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs" pod "393374a1-9e7d-4867-a325-7dfd58126370" (UID: "393374a1-9e7d-4867-a325-7dfd58126370"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:30.033924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.033906 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls" pod "393374a1-9e7d-4867-a325-7dfd58126370" (UID: "393374a1-9e7d-4867-a325-7dfd58126370"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:30.212886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.209777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:30.233174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.233141 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:30.233438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.233411 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393374a1-9e7d-4867-a325-7dfd58126370-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:30.233599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.233576 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znqj\" (UniqueName: \"kubernetes.io/projected/393374a1-9e7d-4867-a325-7dfd58126370-kube-api-access-6znqj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:30.332158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.332127 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:30.817515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.817485 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" Oct 30 19:39:30.817941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.817513 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65" event={"ID":"393374a1-9e7d-4867-a325-7dfd58126370","Type":"ContainerDied","Data":"d563f2a4196ce7a5c1ff5689dbebefe266a10f20021d9c40d53062105ef378f5"} Oct 30 19:39:30.817941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.817572 6514 scope.go:122] "RemoveContainer" containerID="79bd292c98af0c9836bb415bb4d415de97597a28b410ebcade083415e743c080" Oct 30 19:39:30.844509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.843484 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65"] Oct 30 19:39:30.850412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.850385 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82012b-64878c9fcb-gvv65"] Oct 30 19:39:30.936752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:30.936698 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:31.672713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:31.672677 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:32.161837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:32.161808 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393374a1-9e7d-4867-a325-7dfd58126370" path="/var/lib/kubelet/pods/393374a1-9e7d-4867-a325-7dfd58126370/volumes" Oct 30 19:39:32.394064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:32.394030 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:39:32.936015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:32.935915 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:34.253667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:34.253610 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:35.601776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:35.601737 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:36.500821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:36.500786 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:37.356298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:37.356267 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:37.503268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:37.503197 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.040264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:38.040228 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.224890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:38.220434 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.423457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:38.423360 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.586892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:38.580100 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.835780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:38.835745 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.986182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:38.986132 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:39.167457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:39.167380 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx"] Oct 30 19:39:39.167700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:39.167673 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" containerID="cri-o://69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2" gracePeriod=30 Oct 30 19:39:39.231298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:39.231260 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:39.824243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:39.824205 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:40.839298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:40.839225 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:41.823776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:41.823739 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:42.655447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:42.655413 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:43.161317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:43.161278 6514 patch_prober.go:29] interesting pod/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:39:43.161528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:43.161354 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:39:43.548615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:43.548569 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:44.449096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.449067 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:44.565132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.565098 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:39:44.661801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.661758 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.661910 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662053 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662105 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662155 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662185 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662220 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662309 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.662428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.662338 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data\") pod \"ab279feb-f6f1-4ea9-912a-8eef55566912\" (UID: \"ab279feb-f6f1-4ea9-912a-8eef55566912\") " Oct 30 19:39:44.663916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.663216 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:44.663916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.663389 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:44.663916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.663652 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:44.664565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.664262 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:44.665045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.665008 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:44.671607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.671555 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:44.675514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.675471 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "kube-api-access-qlg75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:44.675759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.675732 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:44.676704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.676682 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token" pod "ab279feb-f6f1-4ea9-912a-8eef55566912" (UID: "ab279feb-f6f1-4ea9-912a-8eef55566912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:44.729614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.729567 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:44.764915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.764883 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.764921 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlg75\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-kube-api-access-qlg75\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.764944 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.764960 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.764978 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.764994 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab279feb-f6f1-4ea9-912a-8eef55566912-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.765010 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.765027 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.765272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.765042 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab279feb-f6f1-4ea9-912a-8eef55566912-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:44.860101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:44.859967 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:45.169456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.169369 6514 generic.go:356] "Generic (PLEG): container finished" podID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerID="be96eb88197a1d4ee1372563fda37ff8b794d5bd5d78e24bb21a1040ccdc51fc" exitCode=0 Oct 30 19:39:45.169628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.169445 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" event={"ID":"aa196ee8-c199-4129-8870-2ab2870a75fd","Type":"ContainerDied","Data":"be96eb88197a1d4ee1372563fda37ff8b794d5bd5d78e24bb21a1040ccdc51fc"} Oct 30 19:39:45.170049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.170023 6514 scope.go:122] "RemoveContainer" containerID="be96eb88197a1d4ee1372563fda37ff8b794d5bd5d78e24bb21a1040ccdc51fc" Oct 30 19:39:45.172077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172053 6514 generic.go:356] "Generic (PLEG): container finished" podID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerID="69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2" exitCode=0 Oct 30 19:39:45.172205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172089 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" event={"ID":"ab279feb-f6f1-4ea9-912a-8eef55566912","Type":"ContainerDied","Data":"69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2"} Oct 30 19:39:45.172205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172109 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" Oct 30 19:39:45.172205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172153 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx" event={"ID":"ab279feb-f6f1-4ea9-912a-8eef55566912","Type":"ContainerDied","Data":"1d84aee151b57da1fe383d229f41e468787bef5477852f1793cad35d8ca83280"} Oct 30 19:39:45.172205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172188 6514 scope.go:122] "RemoveContainer" containerID="69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2" Oct 30 19:39:45.172432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172194 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:45.172432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172280 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:45.172432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.172328 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:45.192485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.192463 6514 scope.go:122] "RemoveContainer" containerID="85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79" Oct 30 19:39:45.202719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.202705 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:45.226226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.226131 6514 scope.go:122] "RemoveContainer" containerID="69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2" Oct 30 19:39:45.228616 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:39:45.226555 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2\": container with ID starting with 69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2 not found: ID does not exist" containerID="69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2" Oct 30 19:39:45.228616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.226593 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2"} err="failed to get container status \"69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2\": rpc error: code = NotFound desc = could not find container \"69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2\": container with ID starting with 69845817a9c75fe53b22811531aa34a3b4286c2b8135f26248bb8f5f0850dba2 not found: ID does not exist" Oct 30 19:39:45.228616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.226620 6514 scope.go:122] "RemoveContainer" containerID="85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79" Oct 30 19:39:45.228616 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:39:45.227297 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79\": container with ID starting with 85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79 not found: ID does not exist" containerID="85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79" Oct 30 19:39:45.228616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.227329 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79"} err="failed to get container status \"85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79\": rpc error: code = NotFound desc = could not find container \"85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79\": container with ID starting with 85941d27eda16d1a0a158e41ff05058a8c05aba4cbd9c103a6b97734997bfd79 not found: ID does not exist" Oct 30 19:39:45.229006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.228983 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx"] Oct 30 19:39:45.233530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.233419 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-82012b-openshift-default-67457b4d8c-fx2xx"] Oct 30 19:39:45.396055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.395981 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:45.519604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.519514 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:45.731416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.724248 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:39:45.851452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:45.851360 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:46.121726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.121626 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:46.169001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.168975 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" path="/var/lib/kubelet/pods/ab279feb-f6f1-4ea9-912a-8eef55566912/volumes" Oct 30 19:39:46.182921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.182846 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" event={"ID":"aa196ee8-c199-4129-8870-2ab2870a75fd","Type":"ContainerStarted","Data":"2a64efa7a634aae49a98cb060d5b775cb1312f539afe430aff7e330788bad4ee"} Oct 30 19:39:46.256750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.256717 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:46.768784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.768751 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:39:46.769940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.769913 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:39:46.770067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:46.769977 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:39:47.639517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:47.639424 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:47.769958 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:47.769928 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:39:47.770334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:47.769981 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:39:48.577716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:48.577683 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:48.769141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:48.769105 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:39:48.769313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:48.769171 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:39:49.455536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:49.455500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:49.783202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:49.776341 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:39:49.783202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:49.776422 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:39:49.783202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:49.777244 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:49.964199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:49.956390 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:50.164521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:50.164441 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:50.301885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:50.299531 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:50.769382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:50.769350 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:39:50.769779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:50.769403 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:39:50.884578 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:39:50.884538 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:39:50.932898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:50.931556 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:51.412894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:51.412818 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:51.769146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:51.769112 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:39:51.769327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:51.769176 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 19:39:52.087344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:52.087267 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:52.775989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:52.775959 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:39:52.776390 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:52.776370 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:39:52.777184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:52.777163 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:39:52.992911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:52.989053 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:54.097228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:54.097196 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:55.050147 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.050111 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:55.186145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.186116 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:55.328226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.328142 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:55.502968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.502935 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz"] Oct 30 19:39:55.503590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.503551 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" containerID="cri-o://ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49" gracePeriod=30 Oct 30 19:39:55.508625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.508599 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm"] Oct 30 19:39:55.508847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.508824 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" containerID="cri-o://96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438" gracePeriod=30 Oct 30 19:39:55.576078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:55.576041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:56.200894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:56.197650 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:56.956112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:56.956076 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:57.741309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:57.741268 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:58.609657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:58.609618 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:58.917071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:58.916995 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:59.361018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:39:59.360990 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.202442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.202409 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm"] Oct 30 19:40:00.203648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.203621 6514 cpu_manager.go:414] "containerMap: removing container" podUID="393374a1-9e7d-4867-a325-7dfd58126370" containerName="maas-api" Oct 30 19:40:00.203648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.203647 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393374a1-9e7d-4867-a325-7dfd58126370" containerName="maas-api" Oct 30 19:40:00.203876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.203665 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" Oct 30 19:40:00.203876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.203674 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" Oct 30 19:40:00.204014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.203976 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="393374a1-9e7d-4867-a325-7dfd58126370" containerName="maas-api" Oct 30 19:40:00.204014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.203997 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" Oct 30 19:40:00.204014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.204011 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" Oct 30 19:40:00.208854 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.208831 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.216569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.216544 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ac37c8-serving-cert\"" Oct 30 19:40:00.221742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.220645 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm"] Oct 30 19:40:00.263878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.263839 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.264022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.263950 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.264022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.264013 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.264097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.264054 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.272105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.272076 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.370708 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.370779 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.370893 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.370955 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:40:00.371205 6514 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-serving-cert: secret "maas-api-e2e-inf-b-ac37c8-serving-cert" not found Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:40:00.371276 6514 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls podName:b7d733ab-4541-4446-b1cf-582f83df6a0c nodeName:}" failed. No retries permitted until 2026-10-30 19:40:00.871252883 +0000 UTC m=+1370.829072092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls") pod "maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" (UID: "b7d733ab-4541-4446-b1cf-582f83df6a0c") : secret "maas-api-e2e-inf-b-ac37c8-serving-cert" not found Oct 30 19:40:00.380234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.375567 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.395971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.385218 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.406892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.403679 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:00.406892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.404760 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.641042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.641011 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.847893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.844071 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.926807 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.926932 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.926984 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927105 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927264 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927308 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927346 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927382 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927495 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket\") pod \"e814ff8d-5637-463d-9171-29c0c699182e\" (UID: \"e814ff8d-5637-463d-9171-29c0c699182e\") " Oct 30 19:40:00.928893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:00.927741 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:01.008937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.004662 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:01.012348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.011678 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.031972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.031889 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.039261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.039104 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.040137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.040099 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:01.040267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.040242 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.041393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.041012 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.042738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.042694 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "kube-api-access-z6gnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.045888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.045848 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.046114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.046089 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gnl\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-kube-api-access-z6gnl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.046190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.046123 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e814ff8d-5637-463d-9171-29c0c699182e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.046190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.046139 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.046190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.046154 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e814ff8d-5637-463d-9171-29c0c699182e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.046190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.046175 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.046394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.046190 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.060115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.060086 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.064064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.064037 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo" pod "e814ff8d-5637-463d-9171-29c0c699182e" (UID: "e814ff8d-5637-463d-9171-29c0c699182e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:01.080055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.080025 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:40:01.128945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.128907 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:01.147658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.147591 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.147793 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.147764 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.147942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.147818 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.147942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.147849 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.148092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.147976 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.148092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148053 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.148261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148131 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.148261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148164 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.148261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148189 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data\") pod \"f3b2a985-fc38-41db-a6c3-0352430f1595\" (UID: \"f3b2a985-fc38-41db-a6c3-0352430f1595\") " Oct 30 19:40:01.148261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148217 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.148592 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148563 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:01.148831 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.148784 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.149070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149045 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.149160 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149134 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.149226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149160 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b2a985-fc38-41db-a6c3-0352430f1595-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.149226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149179 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e814ff8d-5637-463d-9171-29c0c699182e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.149226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149193 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e814ff8d-5637-463d-9171-29c0c699182e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.149226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149208 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.149226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.149209 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.152466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.151923 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.152466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.152278 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.155224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.155197 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:01.156472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.156447 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr" pod "f3b2a985-fc38-41db-a6c3-0352430f1595" (UID: "f3b2a985-fc38-41db-a6c3-0352430f1595"). InnerVolumeSpecName "kube-api-access-fgmfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.251514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.251436 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmfr\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-kube-api-access-fgmfr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.251514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.251518 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.252080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.251533 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.252080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.251547 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.252080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.251561 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b2a985-fc38-41db-a6c3-0352430f1595-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.252080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.251574 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b2a985-fc38-41db-a6c3-0352430f1595-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.342270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.342185 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:01.444646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.438805 6514 generic.go:356] "Generic (PLEG): container finished" podID="e814ff8d-5637-463d-9171-29c0c699182e" containerID="ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49" exitCode=0 Oct 30 19:40:01.444646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.439130 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" event={"ID":"e814ff8d-5637-463d-9171-29c0c699182e","Type":"ContainerDied","Data":"ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49"} Oct 30 19:40:01.444646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.439182 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" event={"ID":"e814ff8d-5637-463d-9171-29c0c699182e","Type":"ContainerDied","Data":"2aa2a579a39c0df57d1e0fd1ae97379db75b8cf9d4da2918eae46a9b965e177d"} Oct 30 19:40:01.444646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.439213 6514 scope.go:122] "RemoveContainer" containerID="ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49" Oct 30 19:40:01.444646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.439389 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz" Oct 30 19:40:01.447385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.447348 6514 generic.go:356] "Generic (PLEG): container finished" podID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerID="96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438" exitCode=0 Oct 30 19:40:01.447526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.447416 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" event={"ID":"f3b2a985-fc38-41db-a6c3-0352430f1595","Type":"ContainerDied","Data":"96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438"} Oct 30 19:40:01.447526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.447465 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" event={"ID":"f3b2a985-fc38-41db-a6c3-0352430f1595","Type":"ContainerDied","Data":"77cec4f25e9cad2cbdd9db12c41800c80b6f16ca0a9f6b0973e9b8b409e50ca5"} Oct 30 19:40:01.447635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.447542 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm" Oct 30 19:40:01.460810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.460788 6514 scope.go:122] "RemoveContainer" containerID="ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49" Oct 30 19:40:01.461155 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:40:01.461133 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49\": container with ID starting with ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49 not found: ID does not exist" containerID="ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49" Oct 30 19:40:01.461214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.461164 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49"} err="failed to get container status \"ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49\": rpc error: code = NotFound desc = could not find container \"ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49\": container with ID starting with ebf6056537442c1a20af9294f6ee305f3fb57dcebae9fe80fd8ea299a6483b49 not found: ID does not exist" Oct 30 19:40:01.461214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.461180 6514 scope.go:122] "RemoveContainer" containerID="96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438" Oct 30 19:40:01.470783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.470767 6514 scope.go:122] "RemoveContainer" containerID="96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438" Oct 30 19:40:01.471202 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:40:01.471173 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438\": container with ID starting with 96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438 not found: ID does not exist" containerID="96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438" Oct 30 19:40:01.471283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.471208 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438"} err="failed to get container status \"96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438\": rpc error: code = NotFound desc = could not find container \"96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438\": container with ID starting with 96ccc43c3856461a9dbfdc24b6554b8bb4a55ef5e98e000b1b8fd75ba877f438 not found: ID does not exist" Oct 30 19:40:01.477452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.477426 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm"] Oct 30 19:40:01.477605 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:40:01.477585 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d733ab_4541_4446_b1cf_582f83df6a0c.slice/crio-601b292a45173bd6c72d2da4898806ce1fbf34c9c557158ac96ba0b325fbdf8d WatchSource:0}: Error finding container 601b292a45173bd6c72d2da4898806ce1fbf34c9c557158ac96ba0b325fbdf8d: Status 404 returned error can't find the container with id 601b292a45173bd6c72d2da4898806ce1fbf34c9c557158ac96ba0b325fbdf8d Oct 30 19:40:01.488976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.488955 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz"] Oct 30 19:40:01.501587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.501559 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-vwbfz"] Oct 30 19:40:01.524111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.524076 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm"] Oct 30 19:40:01.532977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:01.532957 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-gspwm"] Oct 30 19:40:02.161657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.161625 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e814ff8d-5637-463d-9171-29c0c699182e" path="/var/lib/kubelet/pods/e814ff8d-5637-463d-9171-29c0c699182e/volumes" Oct 30 19:40:02.162213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.162199 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" path="/var/lib/kubelet/pods/f3b2a985-fc38-41db-a6c3-0352430f1595/volumes" Oct 30 19:40:02.203889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.203305 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:02.458431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.458350 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" event={"ID":"b7d733ab-4541-4446-b1cf-582f83df6a0c","Type":"ContainerStarted","Data":"3ced43fda55c066780feb5e8b63784f117403fd4015e00fa8268d681390cdd9e"} Oct 30 19:40:02.458431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.458404 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" event={"ID":"b7d733ab-4541-4446-b1cf-582f83df6a0c","Type":"ContainerStarted","Data":"601b292a45173bd6c72d2da4898806ce1fbf34c9c557158ac96ba0b325fbdf8d"} Oct 30 19:40:02.458431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.458427 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:02.480376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.480322 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" podStartSLOduration=2.042204021 podStartE2EDuration="2.480306952s" totalImagesPullingTime="438.102931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:02.47895646 +0000 UTC m=+1372.436775650" watchObservedRunningTime="2026-10-30 19:40:02.480306952 +0000 UTC m=+1372.438126192" Oct 30 19:40:02.902571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:02.902491 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:03.665330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:03.665303 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:04.471891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:04.467376 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.149142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:05.149108 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.838001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:05.837930 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:06.575667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:06.575629 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:07.258512 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:07.258474 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:08.016212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:08.016180 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:08.472454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:08.472422 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:40:08.818963 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:08.818849 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:09.462304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:09.462275 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:10.158758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:10.158715 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:10.832954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:10.832919 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:11.310419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.310383 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5"] Oct 30 19:40:11.310656 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.310630 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" podUID="ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" containerName="maas-api" containerID="cri-o://f8cf93e5c7622ddb0a0aea418341c44a5deff495c1fe3b8aa7bb0e2f2e641d74" gracePeriod=30 Oct 30 19:40:11.389762 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.389727 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:11.686932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.661532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:11.808907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.760295 6514 generic.go:356] "Generic (PLEG): container finished" podID="ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" containerID="f8cf93e5c7622ddb0a0aea418341c44a5deff495c1fe3b8aa7bb0e2f2e641d74" exitCode=0 Oct 30 19:40:11.832114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.760327 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" event={"ID":"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245","Type":"ContainerDied","Data":"f8cf93e5c7622ddb0a0aea418341c44a5deff495c1fe3b8aa7bb0e2f2e641d74"} Oct 30 19:40:11.888453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.887409 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:40:11.969317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.969218 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:11.997016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.996984 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca\") pod \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " Oct 30 19:40:11.997173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.997077 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs\") pod \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " Oct 30 19:40:11.997305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.997285 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5\" (UniqueName: \"kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5\") pod \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " Oct 30 19:40:11.997438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.997366 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca" pod "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" (UID: "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:11.997438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.997398 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls\") pod \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\" (UID: \"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245\") " Oct 30 19:40:11.998163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.998137 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:11.999394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.999366 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls" pod "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" (UID: "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:11.999513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.999388 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs" pod "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" (UID: "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:11.999513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:11.999430 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5" pod "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" (UID: "ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245"). InnerVolumeSpecName "kube-api-access-frwd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:12.113599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.106648 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwd5\" (UniqueName: \"kubernetes.io/projected/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-kube-api-access-frwd5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:12.113599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.106741 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:12.113599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.106791 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:12.128885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.127907 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:12.273666 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.273576 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:12.846942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.846902 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" event={"ID":"ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245","Type":"ContainerDied","Data":"e4eb6ac059ce2f5ac14a321dfed678ecba26695b9917f22d5f9cc44177e048b5"} Oct 30 19:40:12.846942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.846944 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5" Oct 30 19:40:12.847248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.846980 6514 scope.go:122] "RemoveContainer" containerID="f8cf93e5c7622ddb0a0aea418341c44a5deff495c1fe3b8aa7bb0e2f2e641d74" Oct 30 19:40:12.903890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.895045 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:12.903890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.901923 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5"] Oct 30 19:40:12.915887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:12.915502 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-156838-74b98c57df-nl9w5"] Oct 30 19:40:13.734103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:13.734068 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.081659 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:14.079314 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.162438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:14.162400 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" path="/var/lib/kubelet/pods/ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245/volumes" Oct 30 19:40:14.221640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:14.221610 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.358349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:14.358267 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.780044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:14.780012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:15.590906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:15.590722 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:15.928306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:15.927988 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:40:17.276711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:17.276679 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.679322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:18.679287 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:19.621761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:19.621722 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.218504 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:20.218475 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.569241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:20.569165 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.882090 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:40:20.882010 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:40:21.392139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:21.392108 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:21.662938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:21.659295 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:21.788306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:21.788268 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.075627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.075594 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh"] Oct 30 19:40:22.076556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076536 6514 cpu_manager.go:414] "containerMap: removing container" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" Oct 30 19:40:22.076556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076556 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" Oct 30 19:40:22.076732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076571 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" Oct 30 19:40:22.076732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076577 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab279feb-f6f1-4ea9-912a-8eef55566912" containerName="istio-proxy" Oct 30 19:40:22.076732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076607 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" Oct 30 19:40:22.076732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076612 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" Oct 30 19:40:22.076732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076633 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" containerName="maas-api" Oct 30 19:40:22.076732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076637 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" containerName="maas-api" Oct 30 19:40:22.077039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076751 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b2a985-fc38-41db-a6c3-0352430f1595" containerName="istio-proxy" Oct 30 19:40:22.077039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076773 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e814ff8d-5637-463d-9171-29c0c699182e" containerName="istio-proxy" Oct 30 19:40:22.077039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.076789 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb64de1-b9b1-49b4-bc78-8cbc6d1b3245" containerName="maas-api" Oct 30 19:40:22.082818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.082795 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.097964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.097940 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-744b57\"/\"openshift-service-ca.crt\"" Oct 30 19:40:22.098135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.098122 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-744b57\"/\"kube-root-ca.crt\"" Oct 30 19:40:22.099387 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.099369 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-744b57\"/\"default-dockercfg-gr98t\"" Oct 30 19:40:22.099516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.099426 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-744b57\"/\"test-model-744b57-kserve-self-signed-certs\"" Oct 30 19:40:22.114685 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.114658 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh"] Oct 30 19:40:22.185280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.185243 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45l4g\" (UniqueName: \"kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.185545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.185516 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.185754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.185624 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.185754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.185684 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.185970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.185809 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.185970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.185890 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.194024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.193996 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.287478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.287448 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45l4g\" (UniqueName: \"kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.287653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.287507 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.287653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.287590 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.287768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.287661 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.287768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.287693 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.287768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.287712 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.288178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.288155 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.288290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.288180 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.288350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.288335 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.290249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.290223 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.290419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.290398 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.318888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.316607 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45l4g\" (UniqueName: \"kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g\") pod \"test-model-744b57-kserve-69d995c749-m7srh\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.358942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.356632 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.401461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.401386 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:22.482606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.482471 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.876144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.876101 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.913736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:22.913710 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh"] Oct 30 19:40:22.915272 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:40:22.915246 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4026156c_0c5f_4d6f_9d9f_5d3fe6a0f980.slice/crio-8bc3ed4e1bd3b2def95f1fc75d9ca096e54acab964a080797270351679710f96 WatchSource:0}: Error finding container 8bc3ed4e1bd3b2def95f1fc75d9ca096e54acab964a080797270351679710f96: Status 404 returned error can't find the container with id 8bc3ed4e1bd3b2def95f1fc75d9ca096e54acab964a080797270351679710f96 Oct 30 19:40:23.000562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.000304 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.145170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.145129 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.205017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.203740 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" event={"ID":"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980","Type":"ContainerStarted","Data":"4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b"} Oct 30 19:40:23.205017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.203807 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" event={"ID":"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980","Type":"ContainerStarted","Data":"8bc3ed4e1bd3b2def95f1fc75d9ca096e54acab964a080797270351679710f96"} Oct 30 19:40:23.230301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.230239 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" podStartSLOduration=1.230222095 podStartE2EDuration="1.230222095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:23.226463151 +0000 UTC m=+1393.184282342" watchObservedRunningTime="2026-10-30 19:40:23.230222095 +0000 UTC m=+1393.188041285" Oct 30 19:40:23.326401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.326348 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.402269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.402233 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:23.468852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.465607 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.586888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.586420 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.720933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:23.715522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:24.735223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:24.735184 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:25.488890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:25.480246 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.533041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:26.533007 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.812148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:26.812064 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.984075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:26.984032 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:27.830653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:27.830617 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:28.148329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:28.148201 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:28.337886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:28.333904 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:28.973407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:28.973368 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.833635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:29.833603 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:30.609430 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:30.609377 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:31.413181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:31.413148 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:32.132582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:32.132548 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:32.292118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:32.292075 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:32.415015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:32.414925 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:32.557060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:32.557021 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:32.940655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:32.940605 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:33.312914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:33.312825 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:40:33.783412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:33.783379 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:34.134343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:34.134255 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:34.244389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:34.244356 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:34.729877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:34.729834 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:35.538513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:35.538477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:36.270073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:36.270042 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:37.009498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:37.009194 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:37.861297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:37.861265 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:38.640981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:38.640943 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:39.343851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:39.343820 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:40.232951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:40.232895 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:40.391749 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:40.391717 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:41.145802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:41.145773 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:42.016924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:42.016884 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:42.863100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:42.863065 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:43.729286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:43.729241 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.382567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:44.382539 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.914727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:44.914695 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.040180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:45.040145 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.189903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:45.184407 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.848632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:45.848596 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.539729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:46.539672 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:47.233673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:47.233617 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:48.495984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:48.495953 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:48.657325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:48.657291 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:49.364030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:49.363990 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.043479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:50.043413 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.733995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:50.733961 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.882750 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:40:50.882713 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:40:51.424008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:51.423976 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:52.138317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:52.138283 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:52.860218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:52.860187 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.896068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:53.896036 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:54.476885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:54.474263 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:40:55.113299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:55.113245 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.283652 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:56.283621 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.972975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:56.972937 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.472642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:57.472612 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.608548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:57.608516 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.950761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:57.950707 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.096883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:58.094536 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.541460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:58.541429 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:59.469045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:40:59.469009 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.264352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:00.264320 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:01.178488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:01.178448 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:01.634444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:01.634406 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:01.758935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:01.758906 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.116595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:02.116535 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.762238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:02.762200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.888401 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:02.888369 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:03.835893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:03.835815 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:04.792492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:04.792453 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.449708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:05.449677 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:06.256068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:06.255990 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.315058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:07.315019 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:08.230640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:08.230601 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:08.962019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:08.961984 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:09.158886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:09.157813 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:09.458051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:09.458010 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:09.770542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:09.770455 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:09.887736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:09.887698 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:10.654536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:10.654500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:11.420889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:11.417800 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:12.174405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:12.174367 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:12.284007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:12.283978 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:12.963796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:12.963767 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:13.659804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:13.659749 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:14.460119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:14.460089 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:15.143081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:15.143035 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.181936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:16.181898 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.323200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:16.323163 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:17.352674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:17.352640 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.307029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:18.306945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.599908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:18.599783 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.649391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:18.649359 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx"] Oct 30 19:41:18.649616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:18.649595 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" containerID="cri-o://2a64efa7a634aae49a98cb060d5b775cb1312f539afe430aff7e330788bad4ee" gracePeriod=30 Oct 30 19:41:18.866938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:18.864848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.810318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:19.810277 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.651325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:20.651293 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.883446 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:41:20.883412 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:41:21.429777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:21.428140 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:22.221896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:22.221514 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:22.778389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:22.778359 6514 patch_prober.go:29] interesting pod/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:41:22.778556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:22.778421 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:41:22.956973 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:22.956926 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.215832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.215798 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6"] Oct 30 19:41:23.221980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.221961 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.236040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.236020 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-744b57-openshift-default-dockercfg-xvhf2\"" Oct 30 19:41:23.239770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.239748 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6"] Oct 30 19:41:23.361598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361563 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.361740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361695 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.361740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361735 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.361834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361799 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.361834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361822 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.361932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361910 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47kgq\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.361981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.361966 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.362036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.362025 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.362086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.362074 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463109 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463148 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463170 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47kgq\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463211 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463248 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463281 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463445 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463547 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463599 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463711 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463796 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.463935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463915 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.464021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.463995 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.464076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.464039 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.465546 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.465523 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.466150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.466102 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.474811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.474788 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.476091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.476068 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47kgq\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.538539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.538515 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:23.945985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.945950 6514 generic.go:356] "Generic (PLEG): container finished" podID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerID="2a64efa7a634aae49a98cb060d5b775cb1312f539afe430aff7e330788bad4ee" exitCode=0 Oct 30 19:41:23.946162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.946019 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" event={"ID":"aa196ee8-c199-4129-8870-2ab2870a75fd","Type":"ContainerDied","Data":"2a64efa7a634aae49a98cb060d5b775cb1312f539afe430aff7e330788bad4ee"} Oct 30 19:41:23.946162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.946081 6514 scope.go:122] "RemoveContainer" containerID="be96eb88197a1d4ee1372563fda37ff8b794d5bd5d78e24bb21a1040ccdc51fc" Oct 30 19:41:23.967618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.967587 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6"] Oct 30 19:41:23.971820 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:41:23.971789 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bf8e4c_e32d_4d8d_8330_cfe10d66c58c.slice/crio-97af401468e6c821c3e0f3cfc59f983a8f36d024c862a4313243ff85ed8621ca WatchSource:0}: Error finding container 97af401468e6c821c3e0f3cfc59f983a8f36d024c862a4313243ff85ed8621ca: Status 404 returned error can't find the container with id 97af401468e6c821c3e0f3cfc59f983a8f36d024c862a4313243ff85ed8621ca Oct 30 19:41:23.974461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.974427 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:23.974589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.974538 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:23.974651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:23.974584 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:24.009511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.009486 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:41:24.049457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.049278 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:24.177284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177255 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177408 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177429 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177543 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177587 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177639 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177684 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177763 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.177841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177762 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:24.177841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.177789 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token\") pod \"aa196ee8-c199-4129-8870-2ab2870a75fd\" (UID: \"aa196ee8-c199-4129-8870-2ab2870a75fd\") " Oct 30 19:41:24.178274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178013 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:24.178274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178232 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:24.178445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178286 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:24.178590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178560 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.178590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178586 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa196ee8-c199-4129-8870-2ab2870a75fd-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.178785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178602 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.178785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178616 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.178785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.178679 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:24.179641 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.179614 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:24.180022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.180001 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:24.180129 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.180082 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:24.180421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.180406 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84" pod "aa196ee8-c199-4129-8870-2ab2870a75fd" (UID: "aa196ee8-c199-4129-8870-2ab2870a75fd"). InnerVolumeSpecName "kube-api-access-cgc84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:24.281017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.280939 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.281017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.280973 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa196ee8-c199-4129-8870-2ab2870a75fd-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.281017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.280989 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgc84\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-kube-api-access-cgc84\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.281017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.281004 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.281483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.281019 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa196ee8-c199-4129-8870-2ab2870a75fd-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.805828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.801613 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:24.959722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.959689 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:24.965263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.965238 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" Oct 30 19:41:24.965413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.965307 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx" event={"ID":"aa196ee8-c199-4129-8870-2ab2870a75fd","Type":"ContainerDied","Data":"a2911fefebf5cc869424ff6e24a36779bd01736bb8c7b07213d6b06afa196b15"} Oct 30 19:41:24.965413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.965371 6514 scope.go:122] "RemoveContainer" containerID="2a64efa7a634aae49a98cb060d5b775cb1312f539afe430aff7e330788bad4ee" Oct 30 19:41:24.967658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.967633 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" event={"ID":"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c","Type":"ContainerStarted","Data":"feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356"} Oct 30 19:41:24.967766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:24.967672 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" event={"ID":"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c","Type":"ContainerStarted","Data":"97af401468e6c821c3e0f3cfc59f983a8f36d024c862a4313243ff85ed8621ca"} Oct 30 19:41:25.021601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:25.021545 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podStartSLOduration=2.021530929 podStartE2EDuration="2.021530929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:25.016429253 +0000 UTC m=+1454.974248445" watchObservedRunningTime="2026-10-30 19:41:25.021530929 +0000 UTC m=+1454.979350118" Oct 30 19:41:25.034134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:25.034104 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx"] Oct 30 19:41:25.040807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:25.040782 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-22af66-openshift-default-5f7c494769-gqftx"] Oct 30 19:41:25.539799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:25.539763 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:25.541271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:25.541245 6514 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:41:25.541399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:25.541300 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:41:26.230009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:26.229968 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" path="/var/lib/kubelet/pods/aa196ee8-c199-4129-8870-2ab2870a75fd/volumes" Oct 30 19:41:26.270887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:26.263703 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.539300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:26.539197 6514 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:41:26.539300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:26.539271 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:41:27.362785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:27.362706 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:27.539097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:27.539054 6514 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:41:27.539253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:27.539118 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:41:28.145956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:28.145927 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:28.539670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:28.539632 6514 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:41:28.540108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:28.539696 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:41:28.974041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:28.974007 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.479210 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:29.479175 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc"] Oct 30 19:41:29.539711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:29.539676 6514 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:41:29.540343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:29.540301 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:41:29.572997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:29.572945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.686594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:29.686543 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:30.267907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:30.266550 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:30.539663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:30.539573 6514 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:41:30.539936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:30.539899 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:41:31.094243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:31.094200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:31.544435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:31.544401 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:31.544918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:31.544892 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:31.545657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:31.545639 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:41:31.813800 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:31.813707 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:32.441018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:32.440981 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.146522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:33.146484 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.244224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.244186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.479714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.479672 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" podUID="30f68ca6-4f5d-4cff-8850-08260c8ede71" containerName="payload-pre-processing" containerID="cri-o://02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a" gracePeriod=25 Oct 30 19:41:34.760118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.760052 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:41:34.866423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.866394 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696\" (UniqueName: \"kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696\") pod \"30f68ca6-4f5d-4cff-8850-08260c8ede71\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " Oct 30 19:41:34.866674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.866519 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume\") pod \"30f68ca6-4f5d-4cff-8850-08260c8ede71\" (UID: \"30f68ca6-4f5d-4cff-8850-08260c8ede71\") " Oct 30 19:41:34.866993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.866966 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume" pod "30f68ca6-4f5d-4cff-8850-08260c8ede71" (UID: "30f68ca6-4f5d-4cff-8850-08260c8ede71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:34.867262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.867242 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f68ca6-4f5d-4cff-8850-08260c8ede71-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:34.868704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.868677 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696" pod "30f68ca6-4f5d-4cff-8850-08260c8ede71" (UID: "30f68ca6-4f5d-4cff-8850-08260c8ede71"). InnerVolumeSpecName "kube-api-access-b7696". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:34.968959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:34.968925 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7696\" (UniqueName: \"kubernetes.io/projected/30f68ca6-4f5d-4cff-8850-08260c8ede71-kube-api-access-b7696\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:35.032211 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.032134 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:35.152627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.152596 6514 generic.go:356] "Generic (PLEG): container finished" podID="30f68ca6-4f5d-4cff-8850-08260c8ede71" containerID="02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a" exitCode=0 Oct 30 19:41:35.152811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.152628 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" Oct 30 19:41:35.152811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.152652 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" event={"ID":"30f68ca6-4f5d-4cff-8850-08260c8ede71","Type":"ContainerDied","Data":"02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a"} Oct 30 19:41:35.152811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.152688 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc" event={"ID":"30f68ca6-4f5d-4cff-8850-08260c8ede71","Type":"ContainerDied","Data":"ee2683e05b95e0290da7c65c9e2d7539cf9b12d53dfca71420624940c842a40a"} Oct 30 19:41:35.152811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.152709 6514 scope.go:122] "RemoveContainer" containerID="02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a" Oct 30 19:41:35.164365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.164348 6514 scope.go:122] "RemoveContainer" containerID="02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a" Oct 30 19:41:35.164642 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:41:35.164618 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a\": container with ID starting with 02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a not found: ID does not exist" containerID="02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a" Oct 30 19:41:35.164724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.164646 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a"} err="failed to get container status \"02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a\": rpc error: code = NotFound desc = could not find container \"02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a\": container with ID starting with 02ca1e00c688cc420e3961fdde2c1073706dcc1ae2809ab0e04ad36c4b30be4a not found: ID does not exist" Oct 30 19:41:35.178034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.178013 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc"] Oct 30 19:41:35.181408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.181384 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-59f017-5fb9d9fcf5-5xlfc"] Oct 30 19:41:35.749191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:35.749160 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.161807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.161731 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f68ca6-4f5d-4cff-8850-08260c8ede71" path="/var/lib/kubelet/pods/30f68ca6-4f5d-4cff-8850-08260c8ede71/volumes" Oct 30 19:41:36.317469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.317431 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd"] Oct 30 19:41:36.317660 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.317626 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" podUID="e8434eb4-8a1e-4d5d-befe-8e13091e911f" containerName="maas-api" containerID="cri-o://327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1" gracePeriod=30 Oct 30 19:41:36.397213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.397186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.646889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.638130 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.765275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.764233 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.888817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.888783 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.947221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:36.946887 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:41:37.072644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.072563 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca\") pod \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " Oct 30 19:41:37.072806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.072644 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq\" (UniqueName: \"kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq\") pod \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " Oct 30 19:41:37.072806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.072770 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls\") pod \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " Oct 30 19:41:37.073053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.072812 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs\") pod \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\" (UID: \"e8434eb4-8a1e-4d5d-befe-8e13091e911f\") " Oct 30 19:41:37.073053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.072883 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca" pod "e8434eb4-8a1e-4d5d-befe-8e13091e911f" (UID: "e8434eb4-8a1e-4d5d-befe-8e13091e911f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:37.073622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.073596 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8434eb4-8a1e-4d5d-befe-8e13091e911f-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:37.075052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.075023 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq" pod "e8434eb4-8a1e-4d5d-befe-8e13091e911f" (UID: "e8434eb4-8a1e-4d5d-befe-8e13091e911f"). InnerVolumeSpecName "kube-api-access-przkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:37.075267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.075248 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs" pod "e8434eb4-8a1e-4d5d-befe-8e13091e911f" (UID: "e8434eb4-8a1e-4d5d-befe-8e13091e911f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:37.075598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.075580 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls" pod "e8434eb4-8a1e-4d5d-befe-8e13091e911f" (UID: "e8434eb4-8a1e-4d5d-befe-8e13091e911f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:37.169403 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.169364 6514 generic.go:356] "Generic (PLEG): container finished" podID="e8434eb4-8a1e-4d5d-befe-8e13091e911f" containerID="327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1" exitCode=0 Oct 30 19:41:37.169604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.169411 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" Oct 30 19:41:37.169604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.169448 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" event={"ID":"e8434eb4-8a1e-4d5d-befe-8e13091e911f","Type":"ContainerDied","Data":"327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1"} Oct 30 19:41:37.169604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.169502 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd" event={"ID":"e8434eb4-8a1e-4d5d-befe-8e13091e911f","Type":"ContainerDied","Data":"5394dbdfcb92e396c7e0cca9e0849b1389cdfccd6fbf98fd214e4adc9dbdc0a1"} Oct 30 19:41:37.169604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.169530 6514 scope.go:122] "RemoveContainer" containerID="327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1" Oct 30 19:41:37.174623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.174600 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:37.174623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.174623 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8434eb4-8a1e-4d5d-befe-8e13091e911f-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:37.174846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.174634 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przkq\" (UniqueName: \"kubernetes.io/projected/e8434eb4-8a1e-4d5d-befe-8e13091e911f-kube-api-access-przkq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:37.180904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.180884 6514 scope.go:122] "RemoveContainer" containerID="327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1" Oct 30 19:41:37.181166 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:41:37.181147 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1\": container with ID starting with 327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1 not found: ID does not exist" containerID="327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1" Oct 30 19:41:37.181220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.181175 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1"} err="failed to get container status \"327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1\": rpc error: code = NotFound desc = could not find container \"327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1\": container with ID starting with 327c6f0147a14ebba65136d347f8c07a4c74b15c0d50a5deca36e88dd60499e1 not found: ID does not exist" Oct 30 19:41:37.195447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.195426 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:37.196293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.196278 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd"] Oct 30 19:41:37.203414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.203384 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-59f017-748f95d546-6bkbd"] Oct 30 19:41:37.309164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.309122 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:37.955021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:37.954427 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:38.160454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.160421 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8434eb4-8a1e-4d5d-befe-8e13091e911f" path="/var/lib/kubelet/pods/e8434eb4-8a1e-4d5d-befe-8e13091e911f/volumes" Oct 30 19:41:38.564960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.564928 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx"] Oct 30 19:41:38.565683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565665 6514 cpu_manager.go:414] "containerMap: removing container" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" Oct 30 19:41:38.565683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565681 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" Oct 30 19:41:38.565879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565726 6514 cpu_manager.go:414] "containerMap: removing container" podUID="30f68ca6-4f5d-4cff-8850-08260c8ede71" containerName="payload-pre-processing" Oct 30 19:41:38.565879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565733 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f68ca6-4f5d-4cff-8850-08260c8ede71" containerName="payload-pre-processing" Oct 30 19:41:38.565879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565749 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e8434eb4-8a1e-4d5d-befe-8e13091e911f" containerName="maas-api" Oct 30 19:41:38.565879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565754 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8434eb4-8a1e-4d5d-befe-8e13091e911f" containerName="maas-api" Oct 30 19:41:38.565879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565848 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" Oct 30 19:41:38.566169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565882 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f68ca6-4f5d-4cff-8850-08260c8ede71" containerName="payload-pre-processing" Oct 30 19:41:38.566169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.565897 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8434eb4-8a1e-4d5d-befe-8e13091e911f" containerName="maas-api" Oct 30 19:41:38.571176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.571156 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.582248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.582225 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3e35df\"/\"openshift-service-ca.crt\"" Oct 30 19:41:38.582370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.582258 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3e35df\"/\"kube-root-ca.crt\"" Oct 30 19:41:38.582370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.582265 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3e35df\"/\"default-dockercfg-d4tbc\"" Oct 30 19:41:38.582370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.582307 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3e35df\"/\"test-model-3e35df-kserve-self-signed-certs\"" Oct 30 19:41:38.594385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.594365 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx"] Oct 30 19:41:38.636912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.634927 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:38.709130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.709092 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.709264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.709161 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.709321 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.709289 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqk9\" (UniqueName: \"kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.709475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.709457 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.709515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.709500 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.709559 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.709524 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.772421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.772389 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:38.812760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.812721 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqk9\" (UniqueName: \"kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.813695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.813668 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.814417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.814398 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.823665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.822187 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.824200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.824179 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.824447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.824428 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.826011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.825986 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.826703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.826384 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.826986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.824826 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.830127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.830106 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.830754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.830687 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqk9\" (UniqueName: \"kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.854529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.847959 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs\") pod \"test-model-3e35df-kserve-66f74fc8dc-h8zmx\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:38.896889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:38.895305 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:39.233852 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:41:39.233812 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4bf65bc_6c93_48e4_941f_a7ac20801141.slice/crio-9c45a8ead6fa316aa096d4f12f77386f79f1fa5a2caac33a53cde3dc777726b2 WatchSource:0}: Error finding container 9c45a8ead6fa316aa096d4f12f77386f79f1fa5a2caac33a53cde3dc777726b2: Status 404 returned error can't find the container with id 9c45a8ead6fa316aa096d4f12f77386f79f1fa5a2caac33a53cde3dc777726b2 Oct 30 19:41:39.234307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:39.234094 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx"] Oct 30 19:41:39.682942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:39.682093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:40.195751 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:40.195711 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" event={"ID":"e4bf65bc-6c93-48e4-941f-a7ac20801141","Type":"ContainerStarted","Data":"70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0"} Oct 30 19:41:40.195934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:40.195766 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" event={"ID":"e4bf65bc-6c93-48e4-941f-a7ac20801141","Type":"ContainerStarted","Data":"9c45a8ead6fa316aa096d4f12f77386f79f1fa5a2caac33a53cde3dc777726b2"} Oct 30 19:41:40.217920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:40.217859 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" podStartSLOduration=2.217844404 podStartE2EDuration="2.217844404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.214944371 +0000 UTC m=+1470.172763561" watchObservedRunningTime="2026-10-30 19:41:40.217844404 +0000 UTC m=+1470.175663590" Oct 30 19:41:40.350120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:40.350084 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:40.896303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:40.896273 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:41.026635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:41.026603 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:41.836698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:41.836665 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.875566 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:42.875533 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.669070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:43.669040 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.786887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:43.780814 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2"] Oct 30 19:41:43.786887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:43.781120 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" containerID="cri-o://3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0" gracePeriod=30 Oct 30 19:41:44.166291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:44.166257 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:41:44.657639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:44.657599 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.647890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:45.647260 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.840900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:45.830703 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.051897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:46.050901 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.378769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:46.378663 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:41:46.458893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:46.457024 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:47.556733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:47.556691 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.348489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:48.348441 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.896031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:48.896000 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:48.909743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:48.909720 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:49.140467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.140441 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:41:49.193148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.193122 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.193333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.193173 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.193333 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.193248 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.193480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.193464 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.193545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.193526 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.193611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.193598 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.194339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194019 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.194339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194043 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.194339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194079 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.194339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194120 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.194339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194176 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs\") pod \"501abc44-e5b5-4b47-bce4-773ccca705e7\" (UID: \"501abc44-e5b5-4b47-bce4-773ccca705e7\") " Oct 30 19:41:49.194339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194258 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:49.196123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194652 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.196123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.194605 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.196533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.196421 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.196533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.196452 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.196533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.196468 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.196533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.196494 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/501abc44-e5b5-4b47-bce4-773ccca705e7-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.196533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.196509 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.197479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.197402 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:49.197792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.197759 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:49.203335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.203310 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "kube-api-access-vpx85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:49.203440 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.203380 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy" pod "501abc44-e5b5-4b47-bce4-773ccca705e7" (UID: "501abc44-e5b5-4b47-bce4-773ccca705e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.218002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.217975 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:49.329889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.315806 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.329889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.315843 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpx85\" (UniqueName: \"kubernetes.io/projected/501abc44-e5b5-4b47-bce4-773ccca705e7-kube-api-access-vpx85\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.329889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.315877 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.329889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.315893 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/501abc44-e5b5-4b47-bce4-773ccca705e7-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.576343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.576306 6514 generic.go:356] "Generic (PLEG): container finished" podID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerID="3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0" exitCode=0 Oct 30 19:41:49.576520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.576358 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" Oct 30 19:41:49.576520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.576390 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" event={"ID":"501abc44-e5b5-4b47-bce4-773ccca705e7","Type":"ContainerDied","Data":"3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0"} Oct 30 19:41:49.576520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.576453 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2" event={"ID":"501abc44-e5b5-4b47-bce4-773ccca705e7","Type":"ContainerDied","Data":"eb43d781d04235cd6e264dbd07d3e4ff0f96afccce9010177e917e339e48372f"} Oct 30 19:41:49.576520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.576485 6514 scope.go:122] "RemoveContainer" containerID="3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0" Oct 30 19:41:49.588511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.588486 6514 scope.go:122] "RemoveContainer" containerID="3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0" Oct 30 19:41:49.588694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.588677 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:41:49.588812 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:41:49.588790 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0\": container with ID starting with 3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0 not found: ID does not exist" containerID="3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0" Oct 30 19:41:49.588885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.588823 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0"} err="failed to get container status \"3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0\": rpc error: code = NotFound desc = could not find container \"3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0\": container with ID starting with 3a2fe5e850a7790f4319b7332c7ec4dcb72e60d5dff53b3d84170708f97690e0 not found: ID does not exist" Oct 30 19:41:49.601520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.601498 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2"] Oct 30 19:41:49.607838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.607817 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-ddzt2"] Oct 30 19:41:49.866201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:49.866132 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.161051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:50.160975 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" path="/var/lib/kubelet/pods/501abc44-e5b5-4b47-bce4-773ccca705e7/volumes" Oct 30 19:41:50.783975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:50.778139 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.883802 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:41:50.883762 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:41:51.467280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:51.467245 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.636220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:51.636185 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.420886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:52.417322 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.203437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:53.203403 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.852228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:53.852194 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.632249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:54.632215 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.508944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:55.508905 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.251472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:56.251437 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.837841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:56.837799 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.948893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:56.948824 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.084237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:57.084198 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.844597 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:57.844562 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:58.665048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:58.665009 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.340218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:59.340173 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.905280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:41:59.905235 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.071886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:00.069140 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.196483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:00.192968 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.366183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:00.366151 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.715031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:00.714994 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.145013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:01.144935 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.946535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:01.946505 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:02.779699 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:02.779667 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.578113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:03.578075 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.356946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:04.356906 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.827116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:04.827075 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.957122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:04.957086 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:05.401395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:05.401362 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.150829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:06.150800 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.830174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:06.830130 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.471549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:07.471516 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.636877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:07.636831 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.840229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:07.840126 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:08.100288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:08.100203 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:08.374555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:08.374469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:08.812962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:08.803645 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:09.806261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:09.806226 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:10.577527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:10.577495 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:10.663132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:10.662205 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:42:10.664746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:10.664723 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:42:11.358793 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:11.358756 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.145663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:12.145634 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.342263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:12.342228 6514 scope.go:122] "RemoveContainer" containerID="5c5d7e323e93bab844c2c3a2f31f757c5be7d2df2db70dcc2a76da35724df290" Oct 30 19:42:12.360311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:12.360287 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.891621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:12.886500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:13.573516 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:13.573485 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:13.690555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:13.690514 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:14.553564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:14.553527 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.346328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:15.346288 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:16.057428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:16.057391 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:16.833697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:16.833667 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:17.526534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:17.526502 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:18.453892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:18.452660 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:18.796005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:18.795931 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44"] Oct 30 19:42:18.806517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:18.806487 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh"] Oct 30 19:42:18.891201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:18.891172 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.008564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:19.008532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.945636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:19.945601 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.867593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:20.867542 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.892116 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:20.892079 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:42:21.575789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:21.575749 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.446649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:22.446613 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.186705 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:23.186656 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.797025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:23.796974 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" podUID="c92245d2-8752-4350-9266-dac03282e97c" containerName="payload-processing" containerID="cri-o://1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146" gracePeriod=25 Oct 30 19:42:23.807420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:23.807386 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerName="payload-pre-processing" containerID="cri-o://778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82" gracePeriod=25 Oct 30 19:42:24.006744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:24.006606 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:24.762281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:24.762244 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.466997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:25.466969 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:26.250750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:26.250718 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:26.933250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:26.933223 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:27.347025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:27.346908 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" podUID="c92245d2-8752-4350-9266-dac03282e97c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:27.347534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:27.346983 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:27.834282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:27.834230 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.541282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:28.541250 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:29.271590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:29.271521 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:30.060415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:30.060365 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:30.974356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:30.974325 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:31.947093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:31.947056 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:32.635506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:32.635473 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:33.325578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:33.325534 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:34.128833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:34.128798 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:34.943513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:34.943465 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.632060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:35.632022 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:36.044876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:36.044838 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:36.200138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:36.200095 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:36.283859 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:36.283827 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:42:36.422026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:36.421131 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:37.091884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:37.086571 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:37.346915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:37.346796 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" podUID="c92245d2-8752-4350-9266-dac03282e97c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:37.346915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:37.346811 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:37.927852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:37.927819 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:38.663096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:38.663039 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.578756 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:39.578718 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:40.476177 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:40.476145 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:41.299717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:41.299678 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:42.150598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:42.150562 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:42.631780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:42.631746 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:42.963037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:42.963006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:43.751011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:43.750969 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.109083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.108971 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:42:44.140365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.140337 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:42:44.144133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.144107 6514 generic.go:356] "Generic (PLEG): container finished" podID="c92245d2-8752-4350-9266-dac03282e97c" containerID="1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146" exitCode=0 Oct 30 19:42:44.144258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.144142 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" Oct 30 19:42:44.144258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.144181 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" event={"ID":"c92245d2-8752-4350-9266-dac03282e97c","Type":"ContainerDied","Data":"1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146"} Oct 30 19:42:44.144258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.144225 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44" event={"ID":"c92245d2-8752-4350-9266-dac03282e97c","Type":"ContainerDied","Data":"82d8babdd38ed54d654dda917a3cefcd604b6010e4e16f2224514d0f3c9b6c12"} Oct 30 19:42:44.144258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.144252 6514 scope.go:122] "RemoveContainer" containerID="1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146" Oct 30 19:42:44.146118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.146095 6514 generic.go:356] "Generic (PLEG): container finished" podID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerID="778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82" exitCode=0 Oct 30 19:42:44.146235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.146186 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" event={"ID":"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9","Type":"ContainerDied","Data":"778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82"} Oct 30 19:42:44.146235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.146218 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" event={"ID":"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9","Type":"ContainerDied","Data":"40690d054d9ba1123a9f7ff3bae6dd83a62623efd156395f7857e29d3666c4f8"} Oct 30 19:42:44.146339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.146274 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh" Oct 30 19:42:44.162615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.162591 6514 scope.go:122] "RemoveContainer" containerID="1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146" Oct 30 19:42:44.163024 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:44.162998 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146\": container with ID starting with 1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146 not found: ID does not exist" containerID="1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146" Oct 30 19:42:44.163128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.163035 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146"} err="failed to get container status \"1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146\": rpc error: code = NotFound desc = could not find container \"1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146\": container with ID starting with 1f4a314945dd2fa00cebd9eeab05bc6479fa410c9a750df71bd4b72b50b07146 not found: ID does not exist" Oct 30 19:42:44.163128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.163065 6514 scope.go:122] "RemoveContainer" containerID="778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82" Oct 30 19:42:44.195857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.195754 6514 scope.go:122] "RemoveContainer" containerID="778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82" Oct 30 19:42:44.196158 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:44.196134 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82\": container with ID starting with 778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82 not found: ID does not exist" containerID="778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82" Oct 30 19:42:44.196244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.196172 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82"} err="failed to get container status \"778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82\": rpc error: code = NotFound desc = could not find container \"778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82\": container with ID starting with 778503cb60c7a493f18dfb881d3c2266dda2e00d37a40ce45999a26e6a35cc82 not found: ID does not exist" Oct 30 19:42:44.223272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.223231 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume\") pod \"c92245d2-8752-4350-9266-dac03282e97c\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " Oct 30 19:42:44.223494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.223476 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7\" (UniqueName: \"kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7\") pod \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " Oct 30 19:42:44.223563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.223546 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume\") pod \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\" (UID: \"f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9\") " Oct 30 19:42:44.223614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.223558 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume" pod "c92245d2-8752-4350-9266-dac03282e97c" (UID: "c92245d2-8752-4350-9266-dac03282e97c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:44.223652 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.223634 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p\" (UniqueName: \"kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p\") pod \"c92245d2-8752-4350-9266-dac03282e97c\" (UID: \"c92245d2-8752-4350-9266-dac03282e97c\") " Oct 30 19:42:44.223957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.223928 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume" pod "f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" (UID: "f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:44.224283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.224261 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92245d2-8752-4350-9266-dac03282e97c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:44.224283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.224283 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:44.225791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.225763 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p" pod "c92245d2-8752-4350-9266-dac03282e97c" (UID: "c92245d2-8752-4350-9266-dac03282e97c"). InnerVolumeSpecName "kube-api-access-9fp2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:44.226221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.226192 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7" pod "f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" (UID: "f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9"). InnerVolumeSpecName "kube-api-access-x47z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:44.325906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.325808 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fp2p\" (UniqueName: \"kubernetes.io/projected/c92245d2-8752-4350-9266-dac03282e97c-kube-api-access-9fp2p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:44.325906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.325835 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47z7\" (UniqueName: \"kubernetes.io/projected/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9-kube-api-access-x47z7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:44.443985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.443950 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.474374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.474343 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44"] Oct 30 19:42:44.481318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.481285 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b623ca-867779ffc6-6dg44"] Oct 30 19:42:44.495352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.495330 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh"] Oct 30 19:42:44.504257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.504235 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b623ca-5c58d97f4c-p6qfh"] Oct 30 19:42:44.635051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.627387 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.762635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.762568 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.963925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:44.961776 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.308954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:45.308837 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.862116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:45.862087 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:46.161644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:46.161560 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92245d2-8752-4350-9266-dac03282e97c" path="/var/lib/kubelet/pods/c92245d2-8752-4350-9266-dac03282e97c/volumes" Oct 30 19:42:46.161999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:46.161982 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" path="/var/lib/kubelet/pods/f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9/volumes" Oct 30 19:42:46.169014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:46.168982 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48"] Oct 30 19:42:46.169170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:46.169150 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" containerID="cri-o://515383a8fa0b6062b5456c1bc1092c8b33d891c5491d6b3acf355708540f77a7" gracePeriod=30 Oct 30 19:42:46.680469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:46.680431 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:47.574175 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:47.574139 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:47.779737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:47.779707 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:48.586715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:48.586677 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:49.509441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:49.509405 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:49.938175 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:49.938141 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.334250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:50.334174 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.641627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:50.641542 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.797871 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:50.797823 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.901611 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:50.901527 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:42:50.954595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:50.954503 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:51.187441 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:51.187401 6514 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263ef9f8_23c6_4bee_ae1f_2dffc850377e.slice/crio-515383a8fa0b6062b5456c1bc1092c8b33d891c5491d6b3acf355708540f77a7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:42:51.310171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.310139 6514 generic.go:356] "Generic (PLEG): container finished" podID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerID="515383a8fa0b6062b5456c1bc1092c8b33d891c5491d6b3acf355708540f77a7" exitCode=0 Oct 30 19:42:51.310332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.310203 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" event={"ID":"263ef9f8-23c6-4bee-ae1f-2dffc850377e","Type":"ContainerDied","Data":"515383a8fa0b6062b5456c1bc1092c8b33d891c5491d6b3acf355708540f77a7"} Oct 30 19:42:51.461226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.461201 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:42:51.552126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552090 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552150 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552225 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552275 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552406 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552415 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:51.552517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552449 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552568 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:51.552669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552579 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:51.552669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552630 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552685 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.552825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552688 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:51.552825 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552762 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy\") pod \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\" (UID: \"263ef9f8-23c6-4bee-ae1f-2dffc850377e\") " Oct 30 19:42:51.553049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.552973 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:51.553428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.553407 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.553535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.553431 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.553535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.553446 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.553535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.553458 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.553535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.553472 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.554690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.554649 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:51.555116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.555079 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "kube-api-access-2s76z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:51.555227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.555180 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:51.555277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.555218 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token" pod "263ef9f8-23c6-4bee-ae1f-2dffc850377e" (UID: "263ef9f8-23c6-4bee-ae1f-2dffc850377e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:51.654607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.654566 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s76z\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-kube-api-access-2s76z\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.654607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.654602 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.654607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.654615 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.654842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.654627 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263ef9f8-23c6-4bee-ae1f-2dffc850377e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.791990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:51.791768 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:52.319181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:52.319149 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" Oct 30 19:42:52.319579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:52.319178 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48" event={"ID":"263ef9f8-23c6-4bee-ae1f-2dffc850377e","Type":"ContainerDied","Data":"c933a2f9a184c5d3cd80ddcc6441ccc7c29241639b4d1405c388e112e28117eb"} Oct 30 19:42:52.319579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:52.319242 6514 scope.go:122] "RemoveContainer" containerID="515383a8fa0b6062b5456c1bc1092c8b33d891c5491d6b3acf355708540f77a7" Oct 30 19:42:52.341923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:52.341900 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48"] Oct 30 19:42:52.350444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:52.350417 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5cd48"] Oct 30 19:42:52.525353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:52.525323 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:53.250138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:53.250107 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:53.409856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:53.409822 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:53.758951 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:53.758913 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:53.919675 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:53.919643 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:54.009881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.009779 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj"] Oct 30 19:42:54.010084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.010058 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" containerID="cri-o://6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa" gracePeriod=30 Oct 30 19:42:54.013527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.013491 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf"] Oct 30 19:42:54.013715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.013690 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" containerID="cri-o://181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432" gracePeriod=30 Oct 30 19:42:54.061313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.061284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:54.262734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.261154 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" path="/var/lib/kubelet/pods/263ef9f8-23c6-4bee-ae1f-2dffc850377e/volumes" Oct 30 19:42:54.339975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:54.339943 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.151224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:55.151190 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.837494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:55.837462 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:56.632241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:56.632200 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:57.283267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:57.283237 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.026352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:58.026275 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.857373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:58.857329 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.347754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.347728 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:42:59.352748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.352720 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:42:59.414904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.414800 6514 generic.go:356] "Generic (PLEG): container finished" podID="80f28040-03ae-4491-b1db-2f289961c491" containerID="6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa" exitCode=0 Oct 30 19:42:59.415064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.414947 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" event={"ID":"80f28040-03ae-4491-b1db-2f289961c491","Type":"ContainerDied","Data":"6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa"} Oct 30 19:42:59.415064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.415001 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" event={"ID":"80f28040-03ae-4491-b1db-2f289961c491","Type":"ContainerDied","Data":"ae4350a03971f2b7c4d7bc4af5698f8cd72f5fd28af1546303e4ac22ccc693be"} Oct 30 19:42:59.415064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.415031 6514 scope.go:122] "RemoveContainer" containerID="6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa" Oct 30 19:42:59.415262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.415200 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj" Oct 30 19:42:59.417471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.417443 6514 generic.go:356] "Generic (PLEG): container finished" podID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerID="181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432" exitCode=0 Oct 30 19:42:59.417574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.417497 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" Oct 30 19:42:59.417574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.417524 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" event={"ID":"40b47461-0713-4d2d-ac1a-4713d1bcfe76","Type":"ContainerDied","Data":"181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432"} Oct 30 19:42:59.417574 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.417571 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf" event={"ID":"40b47461-0713-4d2d-ac1a-4713d1bcfe76","Type":"ContainerDied","Data":"41e88f52cc6f8499406d4d06e9c06311c15586f449063be2bd78d1dbef4b761f"} Oct 30 19:42:59.430212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.430191 6514 scope.go:122] "RemoveContainer" containerID="6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa" Oct 30 19:42:59.430648 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:59.430619 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa\": container with ID starting with 6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa not found: ID does not exist" containerID="6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa" Oct 30 19:42:59.430748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.430660 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa"} err="failed to get container status \"6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa\": rpc error: code = NotFound desc = could not find container \"6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa\": container with ID starting with 6c6e6d25f61fd986714fc5ba5ddd2026545a25e5cd78f2a33300d33dabd245fa not found: ID does not exist" Oct 30 19:42:59.430748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.430685 6514 scope.go:122] "RemoveContainer" containerID="181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432" Oct 30 19:42:59.443068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.443048 6514 scope.go:122] "RemoveContainer" containerID="181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432" Oct 30 19:42:59.443367 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:42:59.443334 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432\": container with ID starting with 181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432 not found: ID does not exist" containerID="181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432" Oct 30 19:42:59.443444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.443378 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432"} err="failed to get container status \"181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432\": rpc error: code = NotFound desc = could not find container \"181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432\": container with ID starting with 181bf066af243e03dcfc11ffc8745b7e4d25cab4af2896aaaff9a9276405c432 not found: ID does not exist" Oct 30 19:42:59.517316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517288 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.517485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517321 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.517485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517354 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.517612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517490 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.517612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517562 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.517717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517612 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.517717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517621 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.517717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517656 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.517717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517628 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.517717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517688 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.517717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517702 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:59.518022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517724 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.518022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517750 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.518022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517780 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.518022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517851 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.518022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517940 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.518022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.517974 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.518391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518033 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.518391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518098 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.518391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518192 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.518391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518227 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4\") pod \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\" (UID: \"40b47461-0713-4d2d-ac1a-4713d1bcfe76\") " Oct 30 19:42:59.518391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518270 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.518391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518366 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs\") pod \"80f28040-03ae-4491-b1db-2f289961c491\" (UID: \"80f28040-03ae-4491-b1db-2f289961c491\") " Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.519314 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.519340 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f28040-03ae-4491-b1db-2f289961c491-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.519355 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.519369 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.519381 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518364 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518834 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518733 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.518883 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.519456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.519191 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.520353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.520305 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.521118 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.521089 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "kube-api-access-tdcp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.521437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.521383 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:59.521556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.521532 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:59.521981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.521950 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy" pod "40b47461-0713-4d2d-ac1a-4713d1bcfe76" (UID: "40b47461-0713-4d2d-ac1a-4713d1bcfe76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.522120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.522094 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.522311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.522290 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.522517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.522499 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk" pod "80f28040-03ae-4491-b1db-2f289961c491" (UID: "80f28040-03ae-4491-b1db-2f289961c491"). InnerVolumeSpecName "kube-api-access-8v2nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.621298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621270 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f28040-03ae-4491-b1db-2f289961c491-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621295 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621298 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621304 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621312 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621324 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621339 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621349 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2nk\" (UniqueName: \"kubernetes.io/projected/80f28040-03ae-4491-b1db-2f289961c491-kube-api-access-8v2nk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621359 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621367 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621375 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b47461-0713-4d2d-ac1a-4713d1bcfe76-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621385 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b47461-0713-4d2d-ac1a-4713d1bcfe76-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621393 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcp4\" (UniqueName: \"kubernetes.io/projected/40b47461-0713-4d2d-ac1a-4713d1bcfe76-kube-api-access-tdcp4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.621492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.621401 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f28040-03ae-4491-b1db-2f289961c491-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.676208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.676149 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.765365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.765334 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj"] Oct 30 19:42:59.770049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.770022 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-smclj"] Oct 30 19:42:59.781764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.781741 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf"] Oct 30 19:42:59.786799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:42:59.786776 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-6z6zf"] Oct 30 19:43:00.160436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:00.160396 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" path="/var/lib/kubelet/pods/40b47461-0713-4d2d-ac1a-4713d1bcfe76/volumes" Oct 30 19:43:00.161148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:00.161125 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f28040-03ae-4491-b1db-2f289961c491" path="/var/lib/kubelet/pods/80f28040-03ae-4491-b1db-2f289961c491/volumes" Oct 30 19:43:00.238826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:00.238793 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:00.356406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:00.356331 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:00.549431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:00.549400 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.040622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:01.040588 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.945904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:01.936599 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:02.780848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:02.780815 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.552319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:03.552287 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:04.564674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:04.564644 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:05.277939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:05.277848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.071910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:06.068008 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.757037 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:06.757002 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:07.597272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:07.597214 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:08.540692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:08.540565 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:09.333033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:09.318566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:09.512270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:09.512237 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.085530 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:10.085488 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.757523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:10.757477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.544957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:11.544926 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:12.268068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:12.268034 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:12.458906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:12.458820 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:13.022388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:13.022356 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:13.227251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:13.227221 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:13.943202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:13.943170 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:14.855189 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:14.855142 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:15.533603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:15.533566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:16.257417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:16.257385 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:17.061454 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:17.061403 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:17.874056 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:17.874025 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:18.757224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:18.757186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:19.457934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:19.457896 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:20.342001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:20.341972 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:20.573750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:20.573715 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:43:20.884306 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:43:20.884273 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:43:21.465663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:21.465629 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:22.273140 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:22.273106 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:23.129039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:23.129002 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:23.771415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:23.771377 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:24.285567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:24.285532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:24.975285 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:24.975256 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:25.964271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:25.964231 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:26.904580 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:26.904529 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:27.775890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:27.772801 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:28.664375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:28.664330 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:29.713066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:29.713020 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.434424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:30.434386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:31.251776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:31.251741 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:32.158537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:32.158482 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:32.918200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:32.918112 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:33.690406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:33.690373 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:34.343603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:34.343562 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:34.662123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:34.662053 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:34.779802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:34.779770 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:35.246584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:35.246555 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:35.936706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:35.936673 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:36.753688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:36.753653 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:37.591589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:37.591546 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:38.557995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:38.557956 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:38.892587 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:38.892503 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh"] Oct 30 19:43:38.949582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:38.949546 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:39.708813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:39.708777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:40.844103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:40.842529 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:41.718885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:41.717465 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.505205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:42.505169 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.669738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:42.669696 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.882528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:42.882433 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:43.085667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:43.085638 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:43.382489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:43.382259 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.071276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:44.071242 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.261906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:44.261855 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:45.020885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:45.018690 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:45.943883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:45.942519 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:46.832378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:46.832343 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:48.445705 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:48.445675 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:49.324824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:49.324785 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl"] Oct 30 19:43:49.478788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:49.478751 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:49.890265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:49.890229 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:50.900933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:50.900116 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:50.947299 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:43:50.947260 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:43:51.810330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:51.810296 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:52.689356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:52.689325 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.394803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:53.394759 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.546891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:53.544724 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.784096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:53.784047 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.931891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:53.931078 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" podUID="4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" containerName="main" containerID="cri-o://4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b" gracePeriod=45 Oct 30 19:43:54.252503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.252477 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:43:54.291557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.291519 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs\" (UniqueName: \"kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs\") pod \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " Oct 30 19:43:54.291780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.291589 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache\") pod \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " Oct 30 19:43:54.291780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.291641 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home\") pod \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " Oct 30 19:43:54.291780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.291723 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir\") pod \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " Oct 30 19:43:54.291780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.291769 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm\") pod \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " Oct 30 19:43:54.292092 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.291858 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g\" (UniqueName: \"kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g\") pod \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\" (UID: \"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980\") " Oct 30 19:43:54.292767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.292720 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache" pod "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" (UID: "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.292993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.292967 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir" pod "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" (UID: "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.294018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.293967 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs" pod "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" (UID: "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:54.294399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.294321 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home" pod "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" (UID: "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.295969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.295939 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g" pod "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" (UID: "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980"). InnerVolumeSpecName "kube-api-access-45l4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:54.296288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.296271 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm" pod "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" (UID: "4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.393738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.393676 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.393738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.393703 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.393738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.393713 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.393738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.393721 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.393738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.393731 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.393738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.393740 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45l4g\" (UniqueName: \"kubernetes.io/projected/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980-kube-api-access-45l4g\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.705544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.705509 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.981552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.981455 6514 generic.go:356] "Generic (PLEG): container finished" podID="4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" containerID="4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b" exitCode=0 Oct 30 19:43:54.981552 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.981508 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" Oct 30 19:43:54.982114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.981541 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" event={"ID":"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980","Type":"ContainerDied","Data":"4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b"} Oct 30 19:43:54.982114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.981614 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh" event={"ID":"4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980","Type":"ContainerDied","Data":"8bc3ed4e1bd3b2def95f1fc75d9ca096e54acab964a080797270351679710f96"} Oct 30 19:43:54.982114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:54.981638 6514 scope.go:122] "RemoveContainer" containerID="4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b" Oct 30 19:43:55.000352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:55.000324 6514 scope.go:122] "RemoveContainer" containerID="4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b" Oct 30 19:43:55.000681 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:43:55.000654 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b\": container with ID starting with 4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b not found: ID does not exist" containerID="4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b" Oct 30 19:43:55.000798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:55.000693 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b"} err="failed to get container status \"4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b\": rpc error: code = NotFound desc = could not find container \"4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b\": container with ID starting with 4a532afc37a3a2d501c26ffcdf6d98759f0ac01bb7084b6b1bef4b4065ee365b not found: ID does not exist" Oct 30 19:43:55.021559 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:55.021532 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh"] Oct 30 19:43:55.042954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:55.042848 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-744b57/test-model-744b57-kserve-69d995c749-m7srh"] Oct 30 19:43:55.141889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:55.139839 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:55.988888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:55.981373 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:56.160594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:56.160560 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" path="/var/lib/kubelet/pods/4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980/volumes" Oct 30 19:43:57.115589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:57.115556 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:58.043887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:58.042487 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:59.026628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:43:59.024717 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:00.225730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:00.225678 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:01.146399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:01.146369 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:02.019885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:02.014687 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:02.857368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:02.857309 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:03.669498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:03.669458 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:03.929480 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:03.929446 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh"] Oct 30 19:44:04.079158 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.079123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:04.346196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.346091 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" podUID="52d0848f-74ed-4310-b6cb-53b7676793e5" containerName="main" containerID="cri-o://0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0" gracePeriod=45 Oct 30 19:44:04.431547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.431517 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d"] Oct 30 19:44:04.654352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.654321 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:44:04.711874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.711826 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache\") pod \"52d0848f-74ed-4310-b6cb-53b7676793e5\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " Oct 30 19:44:04.715976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.715788 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4\" (UniqueName: \"kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4\") pod \"52d0848f-74ed-4310-b6cb-53b7676793e5\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " Oct 30 19:44:04.717322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.712257 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache" pod "52d0848f-74ed-4310-b6cb-53b7676793e5" (UID: "52d0848f-74ed-4310-b6cb-53b7676793e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.717322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.717283 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir\") pod \"52d0848f-74ed-4310-b6cb-53b7676793e5\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " Oct 30 19:44:04.717583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.717413 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home\") pod \"52d0848f-74ed-4310-b6cb-53b7676793e5\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " Oct 30 19:44:04.717583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.717483 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm\") pod \"52d0848f-74ed-4310-b6cb-53b7676793e5\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " Oct 30 19:44:04.717583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.717555 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs\") pod \"52d0848f-74ed-4310-b6cb-53b7676793e5\" (UID: \"52d0848f-74ed-4310-b6cb-53b7676793e5\") " Oct 30 19:44:04.718776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.718144 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home" pod "52d0848f-74ed-4310-b6cb-53b7676793e5" (UID: "52d0848f-74ed-4310-b6cb-53b7676793e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.718776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.718542 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.718776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.718564 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.719113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.719068 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir" pod "52d0848f-74ed-4310-b6cb-53b7676793e5" (UID: "52d0848f-74ed-4310-b6cb-53b7676793e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.721898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.721822 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4" pod "52d0848f-74ed-4310-b6cb-53b7676793e5" (UID: "52d0848f-74ed-4310-b6cb-53b7676793e5"). InnerVolumeSpecName "kube-api-access-hwqw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:04.755088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.730057 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm" pod "52d0848f-74ed-4310-b6cb-53b7676793e5" (UID: "52d0848f-74ed-4310-b6cb-53b7676793e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:04.758151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.757985 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs" pod "52d0848f-74ed-4310-b6cb-53b7676793e5" (UID: "52d0848f-74ed-4310-b6cb-53b7676793e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:04.805923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.805855 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:04.820055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.820018 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqw4\" (UniqueName: \"kubernetes.io/projected/52d0848f-74ed-4310-b6cb-53b7676793e5-kube-api-access-hwqw4\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.820055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.820054 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.820222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.820070 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d0848f-74ed-4310-b6cb-53b7676793e5-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.820222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:04.820085 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d0848f-74ed-4310-b6cb-53b7676793e5-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:05.015886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.014068 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.168643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.163050 6514 generic.go:356] "Generic (PLEG): container finished" podID="52d0848f-74ed-4310-b6cb-53b7676793e5" containerID="0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0" exitCode=0 Oct 30 19:44:05.168643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.163178 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" Oct 30 19:44:05.168643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.163265 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" event={"ID":"52d0848f-74ed-4310-b6cb-53b7676793e5","Type":"ContainerDied","Data":"0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0"} Oct 30 19:44:05.168643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.163302 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl" event={"ID":"52d0848f-74ed-4310-b6cb-53b7676793e5","Type":"ContainerDied","Data":"5e89f9bc35e7f22266a75745eb434c1dfcc0836a63ad2e119b73996d16e701ea"} Oct 30 19:44:05.168643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.163326 6514 scope.go:122] "RemoveContainer" containerID="0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0" Oct 30 19:44:05.191152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.191025 6514 scope.go:122] "RemoveContainer" containerID="0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0" Oct 30 19:44:05.191684 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:05.191604 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0\": container with ID starting with 0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0 not found: ID does not exist" containerID="0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0" Oct 30 19:44:05.191684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.191644 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0"} err="failed to get container status \"0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0\": rpc error: code = NotFound desc = could not find container \"0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0\": container with ID starting with 0895e987a75bb03568aac7afc5ee98f37932ccad949af5f3595216b19b0a85d0 not found: ID does not exist" Oct 30 19:44:05.203382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.201082 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.233282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.233241 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl"] Oct 30 19:44:05.238453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.238427 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-89a3f6/test-model-89a3f6-kserve-6456cdf69c-dk7xl"] Oct 30 19:44:05.629998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.629961 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.761264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:05.761217 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:06.178339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:06.178296 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d0848f-74ed-4310-b6cb-53b7676793e5" path="/var/lib/kubelet/pods/52d0848f-74ed-4310-b6cb-53b7676793e5/volumes" Oct 30 19:44:06.228731 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:06.228703 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:06.437342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:06.437311 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:06.597428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:06.597359 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:06.780319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:06.780246 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:07.306485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:07.306445 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:07.696349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:07.696313 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:08.623961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:08.613246 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:09.431995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.431907 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" podUID="3531267e-df22-4f96-90a8-11244e83f68d" containerName="payload-processing" containerID="cri-o://4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d" gracePeriod=25 Oct 30 19:44:09.708957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.708930 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:44:09.764397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.764360 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:09.855849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.855819 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx\" (UniqueName: \"kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx\") pod \"3531267e-df22-4f96-90a8-11244e83f68d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " Oct 30 19:44:09.856064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.856046 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume\") pod \"3531267e-df22-4f96-90a8-11244e83f68d\" (UID: \"3531267e-df22-4f96-90a8-11244e83f68d\") " Oct 30 19:44:09.856484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.856457 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume" pod "3531267e-df22-4f96-90a8-11244e83f68d" (UID: "3531267e-df22-4f96-90a8-11244e83f68d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:09.856667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.856648 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531267e-df22-4f96-90a8-11244e83f68d-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:09.858151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.858124 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx" pod "3531267e-df22-4f96-90a8-11244e83f68d" (UID: "3531267e-df22-4f96-90a8-11244e83f68d"). InnerVolumeSpecName "kube-api-access-4r8jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:09.957826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:09.957764 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r8jx\" (UniqueName: \"kubernetes.io/projected/3531267e-df22-4f96-90a8-11244e83f68d-kube-api-access-4r8jx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:10.213191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.213115 6514 generic.go:356] "Generic (PLEG): container finished" podID="3531267e-df22-4f96-90a8-11244e83f68d" containerID="4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d" exitCode=0 Oct 30 19:44:10.213191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.213156 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" Oct 30 19:44:10.213383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.213198 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" event={"ID":"3531267e-df22-4f96-90a8-11244e83f68d","Type":"ContainerDied","Data":"4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d"} Oct 30 19:44:10.213383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.213258 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d" event={"ID":"3531267e-df22-4f96-90a8-11244e83f68d","Type":"ContainerDied","Data":"df1b2cfdd624c4e51d8bebf6921dcb9e921bd97074b06233b1ffeff8d0f58fad"} Oct 30 19:44:10.213383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.213293 6514 scope.go:122] "RemoveContainer" containerID="4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d" Oct 30 19:44:10.224279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.224248 6514 scope.go:122] "RemoveContainer" containerID="4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d" Oct 30 19:44:10.224529 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:10.224510 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d\": container with ID starting with 4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d not found: ID does not exist" containerID="4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d" Oct 30 19:44:10.224616 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.224539 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d"} err="failed to get container status \"4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d\": rpc error: code = NotFound desc = could not find container \"4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d\": container with ID starting with 4981a37af541caf97deccd2eb70d9f49ab3d31efb005de5ee4d533012307bf8d not found: ID does not exist" Oct 30 19:44:10.240991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.240966 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d"] Oct 30 19:44:10.250410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.250385 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1e13df-68d544b494-c828d"] Oct 30 19:44:10.517249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.517171 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:44:10.732024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:10.731752 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:12.159507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:12.159476 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3531267e-df22-4f96-90a8-11244e83f68d" path="/var/lib/kubelet/pods/3531267e-df22-4f96-90a8-11244e83f68d/volumes" Oct 30 19:44:12.320534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:12.320497 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:12.557523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:12.557488 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:13.456811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:13.456776 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:14.423847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:14.423803 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:14.470288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:14.470255 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4"] Oct 30 19:44:14.641561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:14.641523 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:14.779886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:14.778361 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:15.428894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:15.427777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.160857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:16.160827 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.510260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:16.506834 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.673500 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:16.673466 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.833050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:16.832969 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:17.466378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:17.466341 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:17.809761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:17.809687 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6"] Oct 30 19:44:17.809950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:17.809933 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" containerID="cri-o://feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356" gracePeriod=30 Oct 30 19:44:17.870484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:17.870449 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:18.046906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:18.037977 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:18.245222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:18.245191 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:18.858932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:18.858898 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g"] Oct 30 19:44:18.859750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:18.859718 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" podUID="6119e345-2afa-428d-9cc6-f1d2f9140389" containerName="maas-api" containerID="cri-o://bf04d3e879d7889c9bfa3c322d582b6abe0c7d5c9bf824d59c2800d07d8c5a52" gracePeriod=30 Oct 30 19:44:18.951106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:18.951061 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" podUID="686d6e36-5155-41cb-9d3d-cb96848abf79" containerName="main" containerID="cri-o://447c0111dff0a6f5faaf72bd4f25a4833131fe097f5e675912007c1c245dd02e" gracePeriod=45 Oct 30 19:44:18.957702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:18.957636 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:19.067713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.067673 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" podUID="686d6e36-5155-41cb-9d3d-cb96848abf79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": dial tcp 10.134.0.147:8000: connect: connection refused" Oct 30 19:44:19.179775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.179746 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:19.358311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.355966 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:19.374584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.373611 6514 generic.go:356] "Generic (PLEG): container finished" podID="6119e345-2afa-428d-9cc6-f1d2f9140389" containerID="bf04d3e879d7889c9bfa3c322d582b6abe0c7d5c9bf824d59c2800d07d8c5a52" exitCode=0 Oct 30 19:44:19.374584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.373757 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" event={"ID":"6119e345-2afa-428d-9cc6-f1d2f9140389","Type":"ContainerDied","Data":"bf04d3e879d7889c9bfa3c322d582b6abe0c7d5c9bf824d59c2800d07d8c5a52"} Oct 30 19:44:19.378940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.376505 6514 generic.go:356] "Generic (PLEG): container finished" podID="686d6e36-5155-41cb-9d3d-cb96848abf79" containerID="447c0111dff0a6f5faaf72bd4f25a4833131fe097f5e675912007c1c245dd02e" exitCode=0 Oct 30 19:44:19.378940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.376557 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" event={"ID":"686d6e36-5155-41cb-9d3d-cb96848abf79","Type":"ContainerDied","Data":"447c0111dff0a6f5faaf72bd4f25a4833131fe097f5e675912007c1c245dd02e"} Oct 30 19:44:19.471271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.471230 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" containerName="payload-processing" containerID="cri-o://209f8e40484aa8ebfd6a8d24e3c32a5a282fce038fc34dd7cae55a36bdea4121" gracePeriod=25 Oct 30 19:44:19.500712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.500683 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:44:19.504710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.504685 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:44:19.530802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.530774 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:19.605584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.605228 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs\" (UniqueName: \"kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs\") pod \"686d6e36-5155-41cb-9d3d-cb96848abf79\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " Oct 30 19:44:19.607573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.607545 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\") pod \"6119e345-2afa-428d-9cc6-f1d2f9140389\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " Oct 30 19:44:19.626610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.626571 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs" pod "686d6e36-5155-41cb-9d3d-cb96848abf79" (UID: "686d6e36-5155-41cb-9d3d-cb96848abf79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:19.626610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.626579 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls" pod "6119e345-2afa-428d-9cc6-f1d2f9140389" (UID: "6119e345-2afa-428d-9cc6-f1d2f9140389"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:19.687101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.687064 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:19.722677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.722604 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr\" (UniqueName: \"kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr\") pod \"6119e345-2afa-428d-9cc6-f1d2f9140389\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " Oct 30 19:44:19.722677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.722651 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache\") pod \"686d6e36-5155-41cb-9d3d-cb96848abf79\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " Oct 30 19:44:19.722935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.722739 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs\") pod \"6119e345-2afa-428d-9cc6-f1d2f9140389\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " Oct 30 19:44:19.722935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.722828 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm\") pod \"686d6e36-5155-41cb-9d3d-cb96848abf79\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " Oct 30 19:44:19.722935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.722879 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca\") pod \"6119e345-2afa-428d-9cc6-f1d2f9140389\" (UID: \"6119e345-2afa-428d-9cc6-f1d2f9140389\") " Oct 30 19:44:19.723139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723033 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache" pod "686d6e36-5155-41cb-9d3d-cb96848abf79" (UID: "686d6e36-5155-41cb-9d3d-cb96848abf79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:19.723139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723042 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home\") pod \"686d6e36-5155-41cb-9d3d-cb96848abf79\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " Oct 30 19:44:19.723248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723150 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r\" (UniqueName: \"kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r\") pod \"686d6e36-5155-41cb-9d3d-cb96848abf79\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " Oct 30 19:44:19.723328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723305 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir\") pod \"686d6e36-5155-41cb-9d3d-cb96848abf79\" (UID: \"686d6e36-5155-41cb-9d3d-cb96848abf79\") " Oct 30 19:44:19.723405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723380 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca" pod "6119e345-2afa-428d-9cc6-f1d2f9140389" (UID: "6119e345-2afa-428d-9cc6-f1d2f9140389"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:19.723627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723560 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir" pod "686d6e36-5155-41cb-9d3d-cb96848abf79" (UID: "686d6e36-5155-41cb-9d3d-cb96848abf79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:19.723722 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.723635 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home" pod "686d6e36-5155-41cb-9d3d-cb96848abf79" (UID: "686d6e36-5155-41cb-9d3d-cb96848abf79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:19.724137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.724116 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686d6e36-5155-41cb-9d3d-cb96848abf79-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.724268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.724141 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.724268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.724159 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6119e345-2afa-428d-9cc6-f1d2f9140389-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.724268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.724175 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.724268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.724188 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.724268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.724201 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.725833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.725797 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs" pod "6119e345-2afa-428d-9cc6-f1d2f9140389" (UID: "6119e345-2afa-428d-9cc6-f1d2f9140389"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:19.726003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.725982 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm" pod "686d6e36-5155-41cb-9d3d-cb96848abf79" (UID: "686d6e36-5155-41cb-9d3d-cb96848abf79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:19.726064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.726012 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr" pod "6119e345-2afa-428d-9cc6-f1d2f9140389" (UID: "6119e345-2afa-428d-9cc6-f1d2f9140389"). InnerVolumeSpecName "kube-api-access-djrvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:19.726064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.726038 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r" pod "686d6e36-5155-41cb-9d3d-cb96848abf79" (UID: "686d6e36-5155-41cb-9d3d-cb96848abf79"). InnerVolumeSpecName "kube-api-access-flm7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:19.854995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.854099 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flm7r\" (UniqueName: \"kubernetes.io/projected/686d6e36-5155-41cb-9d3d-cb96848abf79-kube-api-access-flm7r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.854995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.854144 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djrvr\" (UniqueName: \"kubernetes.io/projected/6119e345-2afa-428d-9cc6-f1d2f9140389-kube-api-access-djrvr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.854995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.854161 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6119e345-2afa-428d-9cc6-f1d2f9140389-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.854995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.854176 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686d6e36-5155-41cb-9d3d-cb96848abf79-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:19.894990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:19.884398 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:20.100738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.100646 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:20.385267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.385180 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" Oct 30 19:44:20.385422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.385178 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g" event={"ID":"6119e345-2afa-428d-9cc6-f1d2f9140389","Type":"ContainerDied","Data":"f62935828af7fd6d1a43a603d68346363c3b6812470ba8e0c8d06cbe9859405c"} Oct 30 19:44:20.385422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.385324 6514 scope.go:122] "RemoveContainer" containerID="bf04d3e879d7889c9bfa3c322d582b6abe0c7d5c9bf824d59c2800d07d8c5a52" Oct 30 19:44:20.386837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.386703 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" Oct 30 19:44:20.386837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.386723 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh" event={"ID":"686d6e36-5155-41cb-9d3d-cb96848abf79","Type":"ContainerDied","Data":"b9147a6283564c98a9376c85ce1afe256471b017a0d3229afad44dd7ae093323"} Oct 30 19:44:20.396083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.396052 6514 scope.go:122] "RemoveContainer" containerID="447c0111dff0a6f5faaf72bd4f25a4833131fe097f5e675912007c1c245dd02e" Oct 30 19:44:20.397420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.397387 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:20.422577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.422548 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g"] Oct 30 19:44:20.434671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.434641 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-89a3f6-5fb866f46c-pxq8g"] Oct 30 19:44:20.464522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.464498 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh"] Oct 30 19:44:20.475293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.475272 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d651c4/test-model-d651c4-kserve-8d886d889-wjxlh"] Oct 30 19:44:20.710492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:20.710455 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:20.882784 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:20.882747 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:44:21.735935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:21.733163 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:22.160397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:22.160294 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6119e345-2afa-428d-9cc6-f1d2f9140389" path="/var/lib/kubelet/pods/6119e345-2afa-428d-9cc6-f1d2f9140389/volumes" Oct 30 19:44:22.160811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:22.160795 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686d6e36-5155-41cb-9d3d-cb96848abf79" path="/var/lib/kubelet/pods/686d6e36-5155-41cb-9d3d-cb96848abf79/volumes" Oct 30 19:44:22.642128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:22.642091 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.093900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.093859 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:44:23.235296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235266 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235364 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235459 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235505 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235562 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:23.235649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235581 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:23.235649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235589 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235616 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235660 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.235853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235794 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:23.236021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235859 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.236021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.235969 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert\") pod \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\" (UID: \"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c\") " Oct 30 19:44:23.236130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236090 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:23.236426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236396 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:23.236677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236654 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.236745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236683 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.236745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236700 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.236745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236714 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.236745 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.236728 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.237837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.237811 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:23.237970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.237935 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "kube-api-access-47kgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:23.238184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.238165 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:23.238265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.238204 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token" pod "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" (UID: "a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:23.338651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.338615 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.338651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.338653 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47kgq\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-kube-api-access-47kgq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.338846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.338673 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.338846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.338687 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:23.418980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.418940 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.460758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.460720 6514 generic.go:356] "Generic (PLEG): container finished" podID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerID="feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356" exitCode=0 Oct 30 19:44:23.461322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.461031 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" Oct 30 19:44:23.462057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.461053 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" event={"ID":"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c","Type":"ContainerDied","Data":"feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356"} Oct 30 19:44:23.462222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.462123 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6" event={"ID":"a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c","Type":"ContainerDied","Data":"97af401468e6c821c3e0f3cfc59f983a8f36d024c862a4313243ff85ed8621ca"} Oct 30 19:44:23.462222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.462169 6514 scope.go:122] "RemoveContainer" containerID="feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356" Oct 30 19:44:23.478971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.478949 6514 scope.go:122] "RemoveContainer" containerID="feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356" Oct 30 19:44:23.479322 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:23.479300 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356\": container with ID starting with feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356 not found: ID does not exist" containerID="feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356" Oct 30 19:44:23.479376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.479333 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356"} err="failed to get container status \"feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356\": rpc error: code = NotFound desc = could not find container \"feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356\": container with ID starting with feffc4d0ce5a50c3980f7758e814fdee4ae7483e5faa405e367e1d75dc502356 not found: ID does not exist" Oct 30 19:44:23.508758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.508710 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6"] Oct 30 19:44:23.530720 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:23.530690 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-b74k6"] Oct 30 19:44:24.160602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:24.160569 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" path="/var/lib/kubelet/pods/a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c/volumes" Oct 30 19:44:24.388887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:24.380544 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:25.123224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:25.123174 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:25.261249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:25.261214 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:25.504914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:25.504853 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:25.752214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:25.752186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:26.270812 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:26.270777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:26.564313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:26.564210 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:27.601252 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:27.601221 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:28.543302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:28.542332 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:28.693490 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:28.693448 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:28.960888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:28.959961 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:29.107204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:29.107169 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:29.287411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:29.287329 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz"] Oct 30 19:44:29.287590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:29.287530 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" containerID="cri-o://efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd" gracePeriod=30 Oct 30 19:44:29.403542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:29.403501 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:29.692887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:29.687189 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:30.397877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:30.397831 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:30.915425 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:30.915386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:30.922277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:30.922105 6514 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:44:30.922696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:30.922663 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:44:31.078817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:31.078784 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l"] Oct 30 19:44:31.308834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:31.308802 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:32.106391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:32.106357 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:33.230889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:33.215993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.104452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.104418 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.561405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.561372 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:44:34.564511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.564487 6514 generic.go:356] "Generic (PLEG): container finished" podID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerID="efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd" exitCode=0 Oct 30 19:44:34.564637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.564522 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" event={"ID":"8ba39fa3-0b84-4a9f-98b2-faad69a3720f","Type":"ContainerDied","Data":"efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd"} Oct 30 19:44:34.564637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.564530 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" Oct 30 19:44:34.564637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.564567 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz" event={"ID":"8ba39fa3-0b84-4a9f-98b2-faad69a3720f","Type":"ContainerDied","Data":"ff9dbd68488ae113d9ae216123652b7d328a1ebed6f5892722c4266139602972"} Oct 30 19:44:34.564637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.564590 6514 scope.go:122] "RemoveContainer" containerID="efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd" Oct 30 19:44:34.578902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.578857 6514 scope.go:122] "RemoveContainer" containerID="61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6" Oct 30 19:44:34.590788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.590753 6514 scope.go:122] "RemoveContainer" containerID="efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd" Oct 30 19:44:34.591356 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:34.591326 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd\": container with ID starting with efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd not found: ID does not exist" containerID="efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd" Oct 30 19:44:34.591467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.591389 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd"} err="failed to get container status \"efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd\": rpc error: code = NotFound desc = could not find container \"efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd\": container with ID starting with efbe980de123a27f8245a9a8f282b9a1a4609aa522ad8cc2a037527f050151fd not found: ID does not exist" Oct 30 19:44:34.591467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.591418 6514 scope.go:122] "RemoveContainer" containerID="61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6" Oct 30 19:44:34.592177 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:34.592141 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6\": container with ID starting with 61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6 not found: ID does not exist" containerID="61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6" Oct 30 19:44:34.592283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.592169 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6"} err="failed to get container status \"61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6\": rpc error: code = NotFound desc = could not find container \"61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6\": container with ID starting with 61e3acae38d98021dd9413c154427f999a1475c4f7d60671f9b2f96a4327a1b6 not found: ID does not exist" Oct 30 19:44:34.684193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684168 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684229 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684288 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684327 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684382 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684426 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684442 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.684556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684458 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.684556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684543 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684584 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684642 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs\") pod \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\" (UID: \"8ba39fa3-0b84-4a9f-98b2-faad69a3720f\") " Oct 30 19:44:34.684810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684704 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:34.685013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.684983 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.685074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.685056 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.685382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.685361 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.685382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.685379 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.685478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.685394 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.685478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.685405 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.685478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.685415 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.687135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.687114 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.687229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.687146 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:34.687229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.687173 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "kube-api-access-xdrqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:34.687311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.687248 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo" pod "8ba39fa3-0b84-4a9f-98b2-faad69a3720f" (UID: "8ba39fa3-0b84-4a9f-98b2-faad69a3720f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:34.786807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.786777 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrqr\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-kube-api-access-xdrqr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.786807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.786807 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.787102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.786817 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.787102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.786826 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba39fa3-0b84-4a9f-98b2-faad69a3720f-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:34.869615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.869580 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.956178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.956102 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz"] Oct 30 19:44:34.958981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:34.958956 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-njwhz"] Oct 30 19:44:35.640311 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:35.640278 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.160520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.160482 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" path="/var/lib/kubelet/pods/8ba39fa3-0b84-4a9f-98b2-faad69a3720f/volumes" Oct 30 19:44:36.260123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.260094 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm"] Oct 30 19:44:36.260328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.260303 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" podUID="b7d733ab-4541-4446-b1cf-582f83df6a0c" containerName="maas-api" containerID="cri-o://3ced43fda55c066780feb5e8b63784f117403fd4015e00fa8268d681390cdd9e" gracePeriod=30 Oct 30 19:44:36.324642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.324607 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.559919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.523497 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph"] Oct 30 19:44:36.637995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.635068 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6"] Oct 30 19:44:36.664888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.661294 6514 generic.go:356] "Generic (PLEG): container finished" podID="b7d733ab-4541-4446-b1cf-582f83df6a0c" containerID="3ced43fda55c066780feb5e8b63784f117403fd4015e00fa8268d681390cdd9e" exitCode=0 Oct 30 19:44:36.664888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.661317 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" event={"ID":"b7d733ab-4541-4446-b1cf-582f83df6a0c","Type":"ContainerDied","Data":"3ced43fda55c066780feb5e8b63784f117403fd4015e00fa8268d681390cdd9e"} Oct 30 19:44:36.668902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.666150 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.733653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.733619 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:44:36.813701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.813659 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.862302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.862267 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs\") pod \"b7d733ab-4541-4446-b1cf-582f83df6a0c\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " Oct 30 19:44:36.862499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.862377 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6\") pod \"b7d733ab-4541-4446-b1cf-582f83df6a0c\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " Oct 30 19:44:36.862582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.862524 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\") pod \"b7d733ab-4541-4446-b1cf-582f83df6a0c\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " Oct 30 19:44:36.862582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.862555 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca\") pod \"b7d733ab-4541-4446-b1cf-582f83df6a0c\" (UID: \"b7d733ab-4541-4446-b1cf-582f83df6a0c\") " Oct 30 19:44:36.863045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.863014 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca" pod "b7d733ab-4541-4446-b1cf-582f83df6a0c" (UID: "b7d733ab-4541-4446-b1cf-582f83df6a0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:36.863205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.863187 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d733ab-4541-4446-b1cf-582f83df6a0c-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:36.864526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.864497 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs" pod "b7d733ab-4541-4446-b1cf-582f83df6a0c" (UID: "b7d733ab-4541-4446-b1cf-582f83df6a0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:36.864526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.864512 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6" pod "b7d733ab-4541-4446-b1cf-582f83df6a0c" (UID: "b7d733ab-4541-4446-b1cf-582f83df6a0c"). InnerVolumeSpecName "kube-api-access-jl2h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:36.864847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.864829 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls" pod "b7d733ab-4541-4446-b1cf-582f83df6a0c" (UID: "b7d733ab-4541-4446-b1cf-582f83df6a0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:36.973960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.973065 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/b7d733ab-4541-4446-b1cf-582f83df6a0c-kube-api-access-jl2h6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:36.973960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.973103 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:36.973960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:36.973115 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d733ab-4541-4446-b1cf-582f83df6a0c-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:37.001886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.001280 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:37.426349 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.426252 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:37.669817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.669785 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" Oct 30 19:44:37.670314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.669816 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm" event={"ID":"b7d733ab-4541-4446-b1cf-582f83df6a0c","Type":"ContainerDied","Data":"601b292a45173bd6c72d2da4898806ce1fbf34c9c557158ac96ba0b325fbdf8d"} Oct 30 19:44:37.670314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.669898 6514 scope.go:122] "RemoveContainer" containerID="3ced43fda55c066780feb5e8b63784f117403fd4015e00fa8268d681390cdd9e" Oct 30 19:44:37.698680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.698654 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm"] Oct 30 19:44:37.706386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.706356 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ac37c8-7fbb6b4c5d-vmsbm"] Oct 30 19:44:37.895309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:37.894239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:38.160372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:38.160340 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d733ab-4541-4446-b1cf-582f83df6a0c" path="/var/lib/kubelet/pods/b7d733ab-4541-4446-b1cf-582f83df6a0c/volumes" Oct 30 19:44:38.896890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:38.890349 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:39.687937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.687904 6514 generic.go:356] "Generic (PLEG): container finished" podID="5f21635e-9c48-466f-9233-3adc4b207c06" containerID="209f8e40484aa8ebfd6a8d24e3c32a5a282fce038fc34dd7cae55a36bdea4121" exitCode=0 Oct 30 19:44:39.688106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.688046 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" event={"ID":"5f21635e-9c48-466f-9233-3adc4b207c06","Type":"ContainerDied","Data":"209f8e40484aa8ebfd6a8d24e3c32a5a282fce038fc34dd7cae55a36bdea4121"} Oct 30 19:44:39.752918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.752886 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:44:39.826121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.826088 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r\" (UniqueName: \"kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r\") pod \"5f21635e-9c48-466f-9233-3adc4b207c06\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " Oct 30 19:44:39.826270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.826164 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume\") pod \"5f21635e-9c48-466f-9233-3adc4b207c06\" (UID: \"5f21635e-9c48-466f-9233-3adc4b207c06\") " Oct 30 19:44:39.826601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.826571 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume" pod "5f21635e-9c48-466f-9233-3adc4b207c06" (UID: "5f21635e-9c48-466f-9233-3adc4b207c06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:39.826796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.826780 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f21635e-9c48-466f-9233-3adc4b207c06-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:39.828253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.828223 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r" pod "5f21635e-9c48-466f-9233-3adc4b207c06" (UID: "5f21635e-9c48-466f-9233-3adc4b207c06"). InnerVolumeSpecName "kube-api-access-qwg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:39.829005 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.828966 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:39.928831 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:39.928785 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwg2r\" (UniqueName: \"kubernetes.io/projected/5f21635e-9c48-466f-9233-3adc4b207c06-kube-api-access-qwg2r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:40.118570 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.118494 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv"] Oct 30 19:44:40.136598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.136566 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9"] Oct 30 19:44:40.219843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.219811 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:40.426932 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.426579 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:40.696779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.696749 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" Oct 30 19:44:40.696992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.696777 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4" event={"ID":"5f21635e-9c48-466f-9233-3adc4b207c06","Type":"ContainerDied","Data":"9cfd52979cadb852d0596eb7bde4ac8769c322928923de3082fe46d49c00baea"} Oct 30 19:44:40.696992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.696845 6514 scope.go:122] "RemoveContainer" containerID="209f8e40484aa8ebfd6a8d24e3c32a5a282fce038fc34dd7cae55a36bdea4121" Oct 30 19:44:40.723417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.723377 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4"] Oct 30 19:44:40.726664 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.726640 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89a3f6-7b8cb866f4-5zqx4"] Oct 30 19:44:40.905845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:40.905686 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:41.560531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:41.560459 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerName="payload-processing" containerID="cri-o://665929b2a5feff16b62cfab37f681fb1c69ba6d45fe06a456121f3ebab986783" gracePeriod=25 Oct 30 19:44:41.598794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:41.598765 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:41.640936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:41.638931 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" containerName="payload-pre-processing" containerID="cri-o://79b5160541ecbb181427132ebce23f45b173467f6d4683dffe498f4d24be5895" gracePeriod=25 Oct 30 19:44:42.140880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:42.140836 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:44:42.160177 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:42.160151 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" path="/var/lib/kubelet/pods/5f21635e-9c48-466f-9233-3adc4b207c06/volumes" Oct 30 19:44:42.634354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:42.633168 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:43.545134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:43.545102 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd"] Oct 30 19:44:43.545355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:43.545331 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" podUID="32e143cc-e88c-417b-9dde-8e6b9310f58d" containerName="maas-api" containerID="cri-o://6f844afdc3bbd1884977b10823dbedae98a504dddd560095d53b1d507fb3de10" gracePeriod=30 Oct 30 19:44:43.709606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:43.709570 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:43.719946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:43.719914 6514 generic.go:356] "Generic (PLEG): container finished" podID="32e143cc-e88c-417b-9dde-8e6b9310f58d" containerID="6f844afdc3bbd1884977b10823dbedae98a504dddd560095d53b1d507fb3de10" exitCode=0 Oct 30 19:44:43.720087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:43.719988 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" event={"ID":"32e143cc-e88c-417b-9dde-8e6b9310f58d","Type":"ContainerDied","Data":"6f844afdc3bbd1884977b10823dbedae98a504dddd560095d53b1d507fb3de10"} Oct 30 19:44:43.952036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:43.948524 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:44:44.017852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.017811 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs\") pod \"32e143cc-e88c-417b-9dde-8e6b9310f58d\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " Oct 30 19:44:44.018071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.017887 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca\") pod \"32e143cc-e88c-417b-9dde-8e6b9310f58d\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " Oct 30 19:44:44.018071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.017964 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\") pod \"32e143cc-e88c-417b-9dde-8e6b9310f58d\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " Oct 30 19:44:44.018071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.018034 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk\" (UniqueName: \"kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk\") pod \"32e143cc-e88c-417b-9dde-8e6b9310f58d\" (UID: \"32e143cc-e88c-417b-9dde-8e6b9310f58d\") " Oct 30 19:44:44.018595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.018566 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca" pod "32e143cc-e88c-417b-9dde-8e6b9310f58d" (UID: "32e143cc-e88c-417b-9dde-8e6b9310f58d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:44.020974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.020941 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk" pod "32e143cc-e88c-417b-9dde-8e6b9310f58d" (UID: "32e143cc-e88c-417b-9dde-8e6b9310f58d"). InnerVolumeSpecName "kube-api-access-mm4tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:44.021150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.021132 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs" pod "32e143cc-e88c-417b-9dde-8e6b9310f58d" (UID: "32e143cc-e88c-417b-9dde-8e6b9310f58d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:44.022267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.022241 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls" pod "32e143cc-e88c-417b-9dde-8e6b9310f58d" (UID: "32e143cc-e88c-417b-9dde-8e6b9310f58d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:44.038077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.035827 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.120525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.120449 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.120525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.120475 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e143cc-e88c-417b-9dde-8e6b9310f58d-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.120525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.120486 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e143cc-e88c-417b-9dde-8e6b9310f58d-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.120525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.120495 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4tk\" (UniqueName: \"kubernetes.io/projected/32e143cc-e88c-417b-9dde-8e6b9310f58d-kube-api-access-mm4tk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.362584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.362554 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.533612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.533582 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.727569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.727539 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" Oct 30 19:44:44.728038 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.727565 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd" event={"ID":"32e143cc-e88c-417b-9dde-8e6b9310f58d","Type":"ContainerDied","Data":"47848dd01160044f86bdbd00783efe7ac36b55ae7cfe1308ee655f5f9481c130"} Oct 30 19:44:44.728038 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.727641 6514 scope.go:122] "RemoveContainer" containerID="6f844afdc3bbd1884977b10823dbedae98a504dddd560095d53b1d507fb3de10" Oct 30 19:44:44.730254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.730220 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.758690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.758661 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd"] Oct 30 19:44:44.768071 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:44.768050 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d651c4-794b8566b4-m77rd"] Oct 30 19:44:45.119523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:45.119480 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerName="payload-processing" containerID="cri-o://15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa" gracePeriod=25 Oct 30 19:44:45.137145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:45.137110 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerName="payload-pre-processing" containerID="cri-o://616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3" gracePeriod=25 Oct 30 19:44:45.268020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:45.267979 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:45.327196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:45.327159 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:45.327371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:45.327228 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:46.100521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.100457 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" containerID="cri-o://f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7" gracePeriod=45 Oct 30 19:44:46.150423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.149848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.176897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.176833 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e143cc-e88c-417b-9dde-8e6b9310f58d" path="/var/lib/kubelet/pods/32e143cc-e88c-417b-9dde-8e6b9310f58d/volumes" Oct 30 19:44:46.250927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.250853 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Oct 30 19:44:46.392683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.392655 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:44:46.488672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.488629 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs\" (UniqueName: \"kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs\") pod \"222cea90-058a-4c0d-96a6-4e870c0b72db\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " Oct 30 19:44:46.488844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.488689 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir\") pod \"222cea90-058a-4c0d-96a6-4e870c0b72db\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " Oct 30 19:44:46.488844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.488763 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz\" (UniqueName: \"kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz\") pod \"222cea90-058a-4c0d-96a6-4e870c0b72db\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " Oct 30 19:44:46.488844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.488814 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm\") pod \"222cea90-058a-4c0d-96a6-4e870c0b72db\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " Oct 30 19:44:46.489030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489006 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir" pod "222cea90-058a-4c0d-96a6-4e870c0b72db" (UID: "222cea90-058a-4c0d-96a6-4e870c0b72db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.489123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489098 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home\") pod \"222cea90-058a-4c0d-96a6-4e870c0b72db\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " Oct 30 19:44:46.489264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489166 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache\") pod \"222cea90-058a-4c0d-96a6-4e870c0b72db\" (UID: \"222cea90-058a-4c0d-96a6-4e870c0b72db\") " Oct 30 19:44:46.489379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489359 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home" pod "222cea90-058a-4c0d-96a6-4e870c0b72db" (UID: "222cea90-058a-4c0d-96a6-4e870c0b72db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.489545 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489518 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache" pod "222cea90-058a-4c0d-96a6-4e870c0b72db" (UID: "222cea90-058a-4c0d-96a6-4e870c0b72db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.489847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489828 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.489930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489857 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.489930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.489910 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.491197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.491174 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm" pod "222cea90-058a-4c0d-96a6-4e870c0b72db" (UID: "222cea90-058a-4c0d-96a6-4e870c0b72db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.491335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.491311 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs" pod "222cea90-058a-4c0d-96a6-4e870c0b72db" (UID: "222cea90-058a-4c0d-96a6-4e870c0b72db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:46.491421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.491400 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz" pod "222cea90-058a-4c0d-96a6-4e870c0b72db" (UID: "222cea90-058a-4c0d-96a6-4e870c0b72db"). InnerVolumeSpecName "kube-api-access-6kbgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:46.590956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.590921 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222cea90-058a-4c0d-96a6-4e870c0b72db-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.590956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.590951 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbgz\" (UniqueName: \"kubernetes.io/projected/222cea90-058a-4c0d-96a6-4e870c0b72db-kube-api-access-6kbgz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.590956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.590961 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222cea90-058a-4c0d-96a6-4e870c0b72db-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.744008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.743972 6514 generic.go:356] "Generic (PLEG): container finished" podID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerID="f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7" exitCode=0 Oct 30 19:44:46.744199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.744026 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" Oct 30 19:44:46.744199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.744041 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" event={"ID":"222cea90-058a-4c0d-96a6-4e870c0b72db","Type":"ContainerDied","Data":"f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7"} Oct 30 19:44:46.744199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.744073 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l" event={"ID":"222cea90-058a-4c0d-96a6-4e870c0b72db","Type":"ContainerDied","Data":"9ea46f80633e90efbbc141dbb6a8b5c86eb5854ecd68a644b127d5b72c02deff"} Oct 30 19:44:46.744199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.744089 6514 scope.go:122] "RemoveContainer" containerID="f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7" Oct 30 19:44:46.755923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.755895 6514 scope.go:122] "RemoveContainer" containerID="f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7" Oct 30 19:44:46.756219 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:46.756195 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7\": container with ID starting with f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7 not found: ID does not exist" containerID="f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7" Oct 30 19:44:46.756274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.756228 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7"} err="failed to get container status \"f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7\": rpc error: code = NotFound desc = could not find container \"f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7\": container with ID starting with f9f93551b7d8d6d250632609bcf9d46c495d6dc562aad59224b38611b815cbf7 not found: ID does not exist" Oct 30 19:44:46.775093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.775053 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l"] Oct 30 19:44:46.787851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:46.787824 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/premium-w6-01b24e-kserve-6d5db8b568-j4x4l"] Oct 30 19:44:47.220380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:47.220342 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:47.515104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:47.515011 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx"] Oct 30 19:44:47.556336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:47.556301 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:47.803647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:47.803566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:47.985514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:47.985474 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.033144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:48.033106 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:48.033328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:48.033176 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:48.160739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:48.160634 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" path="/var/lib/kubelet/pods/222cea90-058a-4c0d-96a6-4e870c0b72db/volumes" Oct 30 19:44:48.166258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:48.166229 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.475903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:48.474826 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.745799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:48.745712 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:49.933522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:49.933492 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:50.912807 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:50.912742 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:44:51.125424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:51.125391 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:52.075141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:52.075103 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.035474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:53.035436 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.435640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:53.435600 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.613394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:53.613362 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.761684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:53.761602 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89"] Oct 30 19:44:53.762161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:53.762127 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" containerID="cri-o://822cae51d84eca02fdad2987056848326db1bf8a197240d0aad3297153c2703c" gracePeriod=30 Oct 30 19:44:53.819093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:53.819059 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:54.061145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:54.061047 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:54.299437 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:54.299395 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:55.171537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:55.171477 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:55.327070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:55.327027 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:55.327254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:55.327027 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:56.016328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:56.016290 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:56.199427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:56.199393 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:56.751441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:56.751408 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g"] Oct 30 19:44:56.751681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:56.751658 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" podUID="abc241c2-1681-4058-97ea-980111f33d4b" containerName="maas-api" containerID="cri-o://0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1" gracePeriod=30 Oct 30 19:44:57.012181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.011708 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:57.027998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.027964 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:44:57.200891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.167068 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv\" (UniqueName: \"kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv\") pod \"abc241c2-1681-4058-97ea-980111f33d4b\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " Oct 30 19:44:57.263323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.263247 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls\") pod \"abc241c2-1681-4058-97ea-980111f33d4b\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " Oct 30 19:44:57.265891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.265377 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca" pod "abc241c2-1681-4058-97ea-980111f33d4b" (UID: "abc241c2-1681-4058-97ea-980111f33d4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:57.275955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.266411 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv" pod "abc241c2-1681-4058-97ea-980111f33d4b" (UID: "abc241c2-1681-4058-97ea-980111f33d4b"). InnerVolumeSpecName "kube-api-access-s7nxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:57.275955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.266673 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca\") pod \"abc241c2-1681-4058-97ea-980111f33d4b\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " Oct 30 19:44:57.275955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.266915 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs\") pod \"abc241c2-1681-4058-97ea-980111f33d4b\" (UID: \"abc241c2-1681-4058-97ea-980111f33d4b\") " Oct 30 19:44:57.275955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.268078 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nxv\" (UniqueName: \"kubernetes.io/projected/abc241c2-1681-4058-97ea-980111f33d4b-kube-api-access-s7nxv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.275955 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.268103 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abc241c2-1681-4058-97ea-980111f33d4b-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.287918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.276457 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs" pod "abc241c2-1681-4058-97ea-980111f33d4b" (UID: "abc241c2-1681-4058-97ea-980111f33d4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:57.287918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.276486 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls" pod "abc241c2-1681-4058-97ea-980111f33d4b" (UID: "abc241c2-1681-4058-97ea-980111f33d4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:57.369458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.369418 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.369458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.369450 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abc241c2-1681-4058-97ea-980111f33d4b-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.403093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.403053 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:57.572777 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.572686 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:57.871653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.871532 6514 generic.go:356] "Generic (PLEG): container finished" podID="abc241c2-1681-4058-97ea-980111f33d4b" containerID="0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1" exitCode=0 Oct 30 19:44:57.871653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.871595 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" Oct 30 19:44:57.871653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.871589 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" event={"ID":"abc241c2-1681-4058-97ea-980111f33d4b","Type":"ContainerDied","Data":"0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1"} Oct 30 19:44:57.871967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.871699 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g" event={"ID":"abc241c2-1681-4058-97ea-980111f33d4b","Type":"ContainerDied","Data":"c109176a2590d8b9881523d1259159d88c667639af38de3ff4ccc994a9f0ff7f"} Oct 30 19:44:57.871967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.871741 6514 scope.go:122] "RemoveContainer" containerID="0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1" Oct 30 19:44:57.886007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.885984 6514 scope.go:122] "RemoveContainer" containerID="0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1" Oct 30 19:44:57.886325 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:44:57.886295 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1\": container with ID starting with 0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1 not found: ID does not exist" containerID="0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1" Oct 30 19:44:57.886400 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.886334 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1"} err="failed to get container status \"0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1\": rpc error: code = NotFound desc = could not find container \"0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1\": container with ID starting with 0f105328370ebac116604c63f970c700412b0c0a2a2352ce5f9df8691dc9aae1 not found: ID does not exist" Oct 30 19:44:57.911369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.911342 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g"] Oct 30 19:44:57.915445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:57.915419 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-01b24e-bb6487c47-vjm8g"] Oct 30 19:44:58.033595 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.033562 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:58.033741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.033589 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:44:58.041216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.041189 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:58.160472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.160385 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc241c2-1681-4058-97ea-980111f33d4b" path="/var/lib/kubelet/pods/abc241c2-1681-4058-97ea-980111f33d4b/volumes" Oct 30 19:44:58.650834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.650805 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:58.813929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.813832 6514 patch_prober.go:29] interesting pod/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": read tcp 10.134.0.2:60430->10.134.0.139:15021: read: connection reset by peer" start-of-body= Oct 30 19:44:58.814459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.814423 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": read tcp 10.134.0.2:60430->10.134.0.139:15021: read: connection reset by peer" Oct 30 19:44:58.853224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.853192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:58.882658 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.882626 6514 generic.go:356] "Generic (PLEG): container finished" podID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerID="822cae51d84eca02fdad2987056848326db1bf8a197240d0aad3297153c2703c" exitCode=0 Oct 30 19:44:58.882807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.882690 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" event={"ID":"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7","Type":"ContainerDied","Data":"822cae51d84eca02fdad2987056848326db1bf8a197240d0aad3297153c2703c"} Oct 30 19:44:58.882807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:58.882749 6514 scope.go:122] "RemoveContainer" containerID="9ef9b1224be0ac04cb6c298089bd3fd1b3fb430e3653b53f41f0c71ed16be8c9" Oct 30 19:44:59.033405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.033378 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:44:59.094688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.094653 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.094858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.094766 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.094858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.094831 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.094940 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.094974 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.095098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.094990 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095060 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095154 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.095332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095165 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095262 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095381 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo\") pod \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\" (UID: \"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7\") " Oct 30 19:44:59.095498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095478 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:59.095629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095604 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.095697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.095672 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.096537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.096504 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.096537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.096531 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.096736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.096547 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.096736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.096562 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.096736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.096577 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.098222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.097939 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.098222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.098060 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "kube-api-access-zjqcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:59.098623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.098592 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:59.099633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.099608 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token" pod "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" (UID: "9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:59.179072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.179039 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc"] Oct 30 19:44:59.197920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.197806 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.197920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.197845 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjqcg\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-kube-api-access-zjqcg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.197920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.197897 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.197920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.197914 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:59.209302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.209264 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4"] Oct 30 19:44:59.342639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.342604 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.343375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.343354 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt"] Oct 30 19:44:59.369258 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.369226 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc"] Oct 30 19:44:59.563359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.563275 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.709923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.698416 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.894330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.894236 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" event={"ID":"9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7","Type":"ContainerDied","Data":"483aad3f2b958cfc10a916a0f3a5d3a2fcfe12a883677f9b3482b9e7a3afeba2"} Oct 30 19:44:59.894330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.894305 6514 scope.go:122] "RemoveContainer" containerID="822cae51d84eca02fdad2987056848326db1bf8a197240d0aad3297153c2703c" Oct 30 19:44:59.894532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.894250 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89" Oct 30 19:44:59.943914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.943847 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89"] Oct 30 19:44:59.946907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:44:59.946844 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d651c4-openshift-default-775869444c-9wh89"] Oct 30 19:45:00.123784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.123740 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:00.161740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.161667 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" path="/var/lib/kubelet/pods/9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7/volumes" Oct 30 19:45:00.174146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174120 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f"] Oct 30 19:45:00.174746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174731 6514 cpu_manager.go:414] "containerMap: removing container" podUID="b7d733ab-4541-4446-b1cf-582f83df6a0c" containerName="maas-api" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174748 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d733ab-4541-4446-b1cf-582f83df6a0c" containerName="maas-api" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174762 6514 cpu_manager.go:414] "containerMap: removing container" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174767 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174774 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174779 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174784 6514 cpu_manager.go:414] "containerMap: removing container" podUID="686d6e36-5155-41cb-9d3d-cb96848abf79" containerName="main" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174789 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686d6e36-5155-41cb-9d3d-cb96848abf79" containerName="main" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174799 6514 cpu_manager.go:414] "containerMap: removing container" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" Oct 30 19:45:00.174803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174804 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174820 6514 cpu_manager.go:414] "containerMap: removing container" podUID="6119e345-2afa-428d-9cc6-f1d2f9140389" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174824 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6119e345-2afa-428d-9cc6-f1d2f9140389" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174837 6514 cpu_manager.go:414] "containerMap: removing container" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174842 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174850 6514 cpu_manager.go:414] "containerMap: removing container" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerName="payload-pre-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174854 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerName="payload-pre-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174877 6514 cpu_manager.go:414] "containerMap: removing container" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174885 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174904 6514 cpu_manager.go:414] "containerMap: removing container" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174909 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174916 6514 cpu_manager.go:414] "containerMap: removing container" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174920 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174927 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c92245d2-8752-4350-9266-dac03282e97c" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174931 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92245d2-8752-4350-9266-dac03282e97c" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174940 6514 cpu_manager.go:414] "containerMap: removing container" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174944 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174953 6514 cpu_manager.go:414] "containerMap: removing container" podUID="abc241c2-1681-4058-97ea-980111f33d4b" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174957 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc241c2-1681-4058-97ea-980111f33d4b" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174965 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174969 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174977 6514 cpu_manager.go:414] "containerMap: removing container" podUID="32e143cc-e88c-417b-9dde-8e6b9310f58d" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.174982 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e143cc-e88c-417b-9dde-8e6b9310f58d" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175004 6514 cpu_manager.go:414] "containerMap: removing container" podUID="3531267e-df22-4f96-90a8-11244e83f68d" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175009 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3531267e-df22-4f96-90a8-11244e83f68d" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175016 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175021 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175028 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175032 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175040 6514 cpu_manager.go:414] "containerMap: removing container" podUID="52d0848f-74ed-4310-b6cb-53b7676793e5" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175044 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d0848f-74ed-4310-b6cb-53b7676793e5" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175054 6514 cpu_manager.go:414] "containerMap: removing container" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175058 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175161 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e143cc-e88c-417b-9dde-8e6b9310f58d" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175170 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="3531267e-df22-4f96-90a8-11244e83f68d" containerName="payload-processing" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175179 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175184 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd983e8-c6f8-4ffd-96ea-64ae8572a0f7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175189 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d0848f-74ed-4310-b6cb-53b7676793e5" containerName="main" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175195 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="501abc44-e5b5-4b47-bce4-773ccca705e7" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175201 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d733ab-4541-4446-b1cf-582f83df6a0c" containerName="maas-api" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175206 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b47461-0713-4d2d-ac1a-4713d1bcfe76" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175212 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" Oct 30 19:45:00.175200 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175217 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="686d6e36-5155-41cb-9d3d-cb96848abf79" containerName="main" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175222 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bf8e4c-e32d-4d8d-8330-cfe10d66c58c" containerName="istio-proxy" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175229 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="6119e345-2afa-428d-9cc6-f1d2f9140389" containerName="maas-api" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175237 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa196ee8-c199-4129-8870-2ab2870a75fd" containerName="istio-proxy" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175243 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01ce9d1-f33c-476a-bd2d-ed66b3b9e2b9" containerName="payload-pre-processing" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175248 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="263ef9f8-23c6-4bee-ae1f-2dffc850377e" containerName="istio-proxy" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175254 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="222cea90-058a-4c0d-96a6-4e870c0b72db" containerName="main" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175259 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f21635e-9c48-466f-9233-3adc4b207c06" containerName="payload-processing" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175265 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92245d2-8752-4350-9266-dac03282e97c" containerName="payload-processing" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175271 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f28040-03ae-4491-b1db-2f289961c491" containerName="istio-proxy" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175277 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc241c2-1681-4058-97ea-980111f33d4b" containerName="maas-api" Oct 30 19:45:00.176510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.175287 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4026156c-0c5f-4d6f-9d9f-5d3fe6a0f980" containerName="main" Oct 30 19:45:00.180223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.180202 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.199428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.199403 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:45:00.206345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.206300 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f"] Oct 30 19:45:00.221699 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.221662 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59"] Oct 30 19:45:00.224414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.223876 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" Oct 30 19:45:00.224414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.223902 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" Oct 30 19:45:00.224414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.224344 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba39fa3-0b84-4a9f-98b2-faad69a3720f" containerName="istio-proxy" Oct 30 19:45:00.239822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.239790 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59"] Oct 30 19:45:00.240124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.240105 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.326295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.326266 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7262z\" (UniqueName: \"kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z\") pod \"maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.326469 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.326371 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.326534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.326506 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswlv\" (UniqueName: \"kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv\") pod \"maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.326583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.326564 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.428226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.428137 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.428226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.428221 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7262z\" (UniqueName: \"kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z\") pod \"maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.428448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.428325 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.428448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.428408 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswlv\" (UniqueName: \"kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv\") pod \"maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.428775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.428748 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.429014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.428991 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.440799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.440776 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswlv\" (UniqueName: \"kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv\") pod \"maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.441047 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.441030 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7262z\" (UniqueName: \"kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z\") pod \"maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.496692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.496665 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:00.560135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.560107 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:00.785913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.785888 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f"] Oct 30 19:45:00.787441 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:45:00.787414 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf20b59_5da1_4cc2_9480_c178b6c94d7b.slice/crio-eb01ffd13b1d35f107277ce00fcddc3f620b45748979942ed71ccb3ea75ad795 WatchSource:0}: Error finding container eb01ffd13b1d35f107277ce00fcddc3f620b45748979942ed71ccb3ea75ad795: Status 404 returned error can't find the container with id eb01ffd13b1d35f107277ce00fcddc3f620b45748979942ed71ccb3ea75ad795 Oct 30 19:45:00.843752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.843722 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm"] Oct 30 19:45:00.844237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.844020 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" podUID="c5c6654f-adc5-4091-a7b2-40af102bb695" containerName="maas-api" containerID="cri-o://2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24" gracePeriod=30 Oct 30 19:45:00.868652 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.868626 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59"] Oct 30 19:45:00.898855 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:45:00.898819 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400fd1c0_4622_455d_abe4_752fded45a70.slice/crio-a319d70e381b7a0c1b8a4b8fce6eb13b2f4948e8b915282db8df93cbf2a2f75d WatchSource:0}: Error finding container a319d70e381b7a0c1b8a4b8fce6eb13b2f4948e8b915282db8df93cbf2a2f75d: Status 404 returned error can't find the container with id a319d70e381b7a0c1b8a4b8fce6eb13b2f4948e8b915282db8df93cbf2a2f75d Oct 30 19:45:00.904760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.904730 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" event={"ID":"eaf20b59-5da1-4cc2-9480-c178b6c94d7b","Type":"ContainerStarted","Data":"eb01ffd13b1d35f107277ce00fcddc3f620b45748979942ed71ccb3ea75ad795"} Oct 30 19:45:00.934548 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:00.934516 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.167969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.156436 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.223494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.223190 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:45:01.296466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.296434 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.342990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.342961 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs\") pod \"c5c6654f-adc5-4091-a7b2-40af102bb695\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " Oct 30 19:45:01.343152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.342998 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\") pod \"c5c6654f-adc5-4091-a7b2-40af102bb695\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " Oct 30 19:45:01.343152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.343097 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk\" (UniqueName: \"kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk\") pod \"c5c6654f-adc5-4091-a7b2-40af102bb695\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " Oct 30 19:45:01.343255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.343180 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca\") pod \"c5c6654f-adc5-4091-a7b2-40af102bb695\" (UID: \"c5c6654f-adc5-4091-a7b2-40af102bb695\") " Oct 30 19:45:01.343507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.343481 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca" pod "c5c6654f-adc5-4091-a7b2-40af102bb695" (UID: "c5c6654f-adc5-4091-a7b2-40af102bb695"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:01.343807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.343790 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c6654f-adc5-4091-a7b2-40af102bb695-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:01.345236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.345195 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk" pod "c5c6654f-adc5-4091-a7b2-40af102bb695" (UID: "c5c6654f-adc5-4091-a7b2-40af102bb695"). InnerVolumeSpecName "kube-api-access-xwcgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:01.345460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.345439 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs" pod "c5c6654f-adc5-4091-a7b2-40af102bb695" (UID: "c5c6654f-adc5-4091-a7b2-40af102bb695"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:01.345460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.345451 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls" pod "c5c6654f-adc5-4091-a7b2-40af102bb695" (UID: "c5c6654f-adc5-4091-a7b2-40af102bb695"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:01.445322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.445283 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:01.445322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.445316 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5c6654f-adc5-4091-a7b2-40af102bb695-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:01.445322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.445327 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcgk\" (UniqueName: \"kubernetes.io/projected/c5c6654f-adc5-4091-a7b2-40af102bb695-kube-api-access-xwcgk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:01.458084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.458055 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.826783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.826755 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.917219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.917186 6514 generic.go:356] "Generic (PLEG): container finished" podID="400fd1c0-4622-455d-abe4-752fded45a70" containerID="d567b8179f75ed541c8c0cba99664523b9ef73d59b9013c47f79b86935ed36f9" exitCode=60 Oct 30 19:45:01.917359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.917317 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" event={"ID":"400fd1c0-4622-455d-abe4-752fded45a70","Type":"ContainerDied","Data":"d567b8179f75ed541c8c0cba99664523b9ef73d59b9013c47f79b86935ed36f9"} Oct 30 19:45:01.917436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.917392 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" event={"ID":"400fd1c0-4622-455d-abe4-752fded45a70","Type":"ContainerStarted","Data":"a319d70e381b7a0c1b8a4b8fce6eb13b2f4948e8b915282db8df93cbf2a2f75d"} Oct 30 19:45:01.918101 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.917596 6514 scope.go:122] "RemoveContainer" containerID="d567b8179f75ed541c8c0cba99664523b9ef73d59b9013c47f79b86935ed36f9" Oct 30 19:45:01.919939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.919914 6514 generic.go:356] "Generic (PLEG): container finished" podID="e15ed13e-ac83-4499-b154-05c89333c29c" containerID="79b5160541ecbb181427132ebce23f45b173467f6d4683dffe498f4d24be5895" exitCode=0 Oct 30 19:45:01.920051 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.920032 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" event={"ID":"e15ed13e-ac83-4499-b154-05c89333c29c","Type":"ContainerDied","Data":"79b5160541ecbb181427132ebce23f45b173467f6d4683dffe498f4d24be5895"} Oct 30 19:45:01.921778 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.921704 6514 generic.go:356] "Generic (PLEG): container finished" podID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerID="665929b2a5feff16b62cfab37f681fb1c69ba6d45fe06a456121f3ebab986783" exitCode=0 Oct 30 19:45:01.921778 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.921743 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" event={"ID":"269a3f6e-538b-4be8-8a51-b056342c28fe","Type":"ContainerDied","Data":"665929b2a5feff16b62cfab37f681fb1c69ba6d45fe06a456121f3ebab986783"} Oct 30 19:45:01.921970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.921810 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" event={"ID":"269a3f6e-538b-4be8-8a51-b056342c28fe","Type":"ContainerDied","Data":"9b8c12a60dfd332def751a3149fcb1e008eabe91d21c95ad1c0bf5267a166e4c"} Oct 30 19:45:01.921970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.921837 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8c12a60dfd332def751a3149fcb1e008eabe91d21c95ad1c0bf5267a166e4c" Oct 30 19:45:01.926797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.926031 6514 generic.go:356] "Generic (PLEG): container finished" podID="c5c6654f-adc5-4091-a7b2-40af102bb695" containerID="2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24" exitCode=0 Oct 30 19:45:01.926797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.926180 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" event={"ID":"c5c6654f-adc5-4091-a7b2-40af102bb695","Type":"ContainerDied","Data":"2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24"} Oct 30 19:45:01.926797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.926218 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" event={"ID":"c5c6654f-adc5-4091-a7b2-40af102bb695","Type":"ContainerDied","Data":"f2ea25fe111da0538c98dbc4b39a49c8483cf8701c2dc868194b64fd842a64c5"} Oct 30 19:45:01.926797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.926247 6514 scope.go:122] "RemoveContainer" containerID="2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24" Oct 30 19:45:01.926797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.926367 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm" Oct 30 19:45:01.934905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.934399 6514 generic.go:356] "Generic (PLEG): container finished" podID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerID="002477cc613b696e8dc073a9e66a3b9ef28cc47bf60d2942213698f61577ded2" exitCode=60 Oct 30 19:45:01.934905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.934470 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" event={"ID":"eaf20b59-5da1-4cc2-9480-c178b6c94d7b","Type":"ContainerDied","Data":"002477cc613b696e8dc073a9e66a3b9ef28cc47bf60d2942213698f61577ded2"} Oct 30 19:45:01.934905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:01.934736 6514 scope.go:122] "RemoveContainer" containerID="002477cc613b696e8dc073a9e66a3b9ef28cc47bf60d2942213698f61577ded2" Oct 30 19:45:02.003159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.003140 6514 scope.go:122] "RemoveContainer" containerID="2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24" Oct 30 19:45:02.003478 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:02.003460 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24\": container with ID starting with 2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24 not found: ID does not exist" containerID="2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24" Oct 30 19:45:02.003523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.003490 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24"} err="failed to get container status \"2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24\": rpc error: code = NotFound desc = could not find container \"2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24\": container with ID starting with 2626ba4ff27c3291735bbdef077a120daed110a142ef9b75d91603a45aafdd24 not found: ID does not exist" Oct 30 19:45:02.015015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.014987 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:45:02.042560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.042532 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:45:02.067152 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.067127 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm"] Oct 30 19:45:02.070263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.070235 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fdb355-6b8565bc64-kqcfm"] Oct 30 19:45:02.154318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.154290 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume\") pod \"e15ed13e-ac83-4499-b154-05c89333c29c\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " Oct 30 19:45:02.154507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.154466 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n\" (UniqueName: \"kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n\") pod \"269a3f6e-538b-4be8-8a51-b056342c28fe\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " Oct 30 19:45:02.154507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.154498 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb\" (UniqueName: \"kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb\") pod \"e15ed13e-ac83-4499-b154-05c89333c29c\" (UID: \"e15ed13e-ac83-4499-b154-05c89333c29c\") " Oct 30 19:45:02.154670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.154655 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume\") pod \"269a3f6e-538b-4be8-8a51-b056342c28fe\" (UID: \"269a3f6e-538b-4be8-8a51-b056342c28fe\") " Oct 30 19:45:02.154783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.154735 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume" pod "e15ed13e-ac83-4499-b154-05c89333c29c" (UID: "e15ed13e-ac83-4499-b154-05c89333c29c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:02.155245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.155220 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume" pod "269a3f6e-538b-4be8-8a51-b056342c28fe" (UID: "269a3f6e-538b-4be8-8a51-b056342c28fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:02.155376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.155245 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ed13e-ac83-4499-b154-05c89333c29c-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.160857 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.160832 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c6654f-adc5-4091-a7b2-40af102bb695" path="/var/lib/kubelet/pods/c5c6654f-adc5-4091-a7b2-40af102bb695/volumes" Oct 30 19:45:02.192380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.192342 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n" pod "269a3f6e-538b-4be8-8a51-b056342c28fe" (UID: "269a3f6e-538b-4be8-8a51-b056342c28fe"). InnerVolumeSpecName "kube-api-access-vfr8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:02.192380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.192365 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb" pod "e15ed13e-ac83-4499-b154-05c89333c29c" (UID: "e15ed13e-ac83-4499-b154-05c89333c29c"). InnerVolumeSpecName "kube-api-access-5vcpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:02.256943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.256878 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfr8n\" (UniqueName: \"kubernetes.io/projected/269a3f6e-538b-4be8-8a51-b056342c28fe-kube-api-access-vfr8n\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.256943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.256913 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vcpb\" (UniqueName: \"kubernetes.io/projected/e15ed13e-ac83-4499-b154-05c89333c29c-kube-api-access-5vcpb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.256943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.256928 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269a3f6e-538b-4be8-8a51-b056342c28fe-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.408764 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.405894 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:02.535999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.535929 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" podUID="e4bf65bc-6c93-48e4-941f-a7ac20801141" containerName="main" containerID="cri-o://70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0" gracePeriod=45 Oct 30 19:45:02.577357 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.577320 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" podUID="e4bf65bc-6c93-48e4-941f-a7ac20801141" containerName="main" probeResult="failure" output="Get \"https://10.134.0.151:8000/ready\": dial tcp 10.134.0.151:8000: connect: connection refused" Oct 30 19:45:02.800653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.800589 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:45:02.864727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864700 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir\") pod \"e4bf65bc-6c93-48e4-941f-a7ac20801141\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " Oct 30 19:45:02.865112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864743 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache\") pod \"e4bf65bc-6c93-48e4-941f-a7ac20801141\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " Oct 30 19:45:02.865112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864769 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9\" (UniqueName: \"kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9\") pod \"e4bf65bc-6c93-48e4-941f-a7ac20801141\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " Oct 30 19:45:02.865112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864795 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs\") pod \"e4bf65bc-6c93-48e4-941f-a7ac20801141\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " Oct 30 19:45:02.865112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864825 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm\") pod \"e4bf65bc-6c93-48e4-941f-a7ac20801141\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " Oct 30 19:45:02.865112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864915 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home\") pod \"e4bf65bc-6c93-48e4-941f-a7ac20801141\" (UID: \"e4bf65bc-6c93-48e4-941f-a7ac20801141\") " Oct 30 19:45:02.865112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.864971 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir" pod "e4bf65bc-6c93-48e4-941f-a7ac20801141" (UID: "e4bf65bc-6c93-48e4-941f-a7ac20801141"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:02.865381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.865115 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache" pod "e4bf65bc-6c93-48e4-941f-a7ac20801141" (UID: "e4bf65bc-6c93-48e4-941f-a7ac20801141"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:02.865381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.865216 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home" pod "e4bf65bc-6c93-48e4-941f-a7ac20801141" (UID: "e4bf65bc-6c93-48e4-941f-a7ac20801141"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:02.865553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.865531 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.865608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.865558 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.865608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.865575 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.867078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.867048 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs" pod "e4bf65bc-6c93-48e4-941f-a7ac20801141" (UID: "e4bf65bc-6c93-48e4-941f-a7ac20801141"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:02.867078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.867065 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm" pod "e4bf65bc-6c93-48e4-941f-a7ac20801141" (UID: "e4bf65bc-6c93-48e4-941f-a7ac20801141"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:02.867206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.867057 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9" pod "e4bf65bc-6c93-48e4-941f-a7ac20801141" (UID: "e4bf65bc-6c93-48e4-941f-a7ac20801141"). InnerVolumeSpecName "kube-api-access-tfqk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:02.946171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.946140 6514 generic.go:356] "Generic (PLEG): container finished" podID="400fd1c0-4622-455d-abe4-752fded45a70" containerID="070e0dc886620d474586ea62aee89e216b9430752687cafde0bb5fac125376ae" exitCode=60 Oct 30 19:45:02.946322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.946284 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" event={"ID":"400fd1c0-4622-455d-abe4-752fded45a70","Type":"ContainerDied","Data":"070e0dc886620d474586ea62aee89e216b9430752687cafde0bb5fac125376ae"} Oct 30 19:45:02.946410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.946367 6514 scope.go:122] "RemoveContainer" containerID="d567b8179f75ed541c8c0cba99664523b9ef73d59b9013c47f79b86935ed36f9" Oct 30 19:45:02.946624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.946585 6514 scope.go:122] "RemoveContainer" containerID="070e0dc886620d474586ea62aee89e216b9430752687cafde0bb5fac125376ae" Oct 30 19:45:02.947288 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:02.947021 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59_odh-ai-gateway-infra(400fd1c0-4622-455d-abe4-752fded45a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" podUID="400fd1c0-4622-455d-abe4-752fded45a70" Oct 30 19:45:02.948114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.948093 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" Oct 30 19:45:02.948219 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.948115 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6" event={"ID":"e15ed13e-ac83-4499-b154-05c89333c29c","Type":"ContainerDied","Data":"fc67856103a9adf9ba8e03d52c3ed6494d57b19a1e518d1d83d23da192ff8ffa"} Oct 30 19:45:02.950505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.950480 6514 generic.go:356] "Generic (PLEG): container finished" podID="e4bf65bc-6c93-48e4-941f-a7ac20801141" containerID="70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0" exitCode=0 Oct 30 19:45:02.950619 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.950606 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" Oct 30 19:45:02.950840 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.950811 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" event={"ID":"e4bf65bc-6c93-48e4-941f-a7ac20801141","Type":"ContainerDied","Data":"70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0"} Oct 30 19:45:02.950987 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.950855 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx" event={"ID":"e4bf65bc-6c93-48e4-941f-a7ac20801141","Type":"ContainerDied","Data":"9c45a8ead6fa316aa096d4f12f77386f79f1fa5a2caac33a53cde3dc777726b2"} Oct 30 19:45:02.953488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.953470 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph" Oct 30 19:45:02.956989 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.956963 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" event={"ID":"eaf20b59-5da1-4cc2-9480-c178b6c94d7b","Type":"ContainerStarted","Data":"775d4a5ddf8be9d15b92499e54a1f9ccc997ab1c0c1a3b3bb628487a2447ab8d"} Oct 30 19:45:02.960712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.960555 6514 scope.go:122] "RemoveContainer" containerID="79b5160541ecbb181427132ebce23f45b173467f6d4683dffe498f4d24be5895" Oct 30 19:45:02.966620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.966598 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqk9\" (UniqueName: \"kubernetes.io/projected/e4bf65bc-6c93-48e4-941f-a7ac20801141-kube-api-access-tfqk9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.966717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.966625 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bf65bc-6c93-48e4-941f-a7ac20801141-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.966717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.966642 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bf65bc-6c93-48e4-941f-a7ac20801141-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.974299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.974282 6514 scope.go:122] "RemoveContainer" containerID="70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0" Oct 30 19:45:02.983524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.983506 6514 scope.go:122] "RemoveContainer" containerID="70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0" Oct 30 19:45:02.983804 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:02.983784 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0\": container with ID starting with 70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0 not found: ID does not exist" containerID="70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0" Oct 30 19:45:02.983880 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.983813 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0"} err="failed to get container status \"70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0\": rpc error: code = NotFound desc = could not find container \"70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0\": container with ID starting with 70148c0db46dcf3897be144aeff9ac2c13d60c8472363df4e8b56494d6c40de0 not found: ID does not exist" Oct 30 19:45:02.998154 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:02.998117 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph"] Oct 30 19:45:03.001819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.001797 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-01b24e-7b4dc88878-4fbph"] Oct 30 19:45:03.010309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.010281 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:45:03.031193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.031167 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx"] Oct 30 19:45:03.039093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.039060 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3e35df/test-model-3e35df-kserve-66f74fc8dc-h8zmx"] Oct 30 19:45:03.068524 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.068469 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6"] Oct 30 19:45:03.070810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.070788 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-01b24e-f7447dc8c-z5lr6"] Oct 30 19:45:03.096849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.096810 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" podStartSLOduration=3.096798582 podStartE2EDuration="3.096798582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.096462226 +0000 UTC m=+1673.054281414" watchObservedRunningTime="2026-10-30 19:45:03.096798582 +0000 UTC m=+1673.054617770" Oct 30 19:45:03.626605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.626565 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:03.971891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.970608 6514 scope.go:122] "RemoveContainer" containerID="070e0dc886620d474586ea62aee89e216b9430752687cafde0bb5fac125376ae" Oct 30 19:45:03.971891 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:03.971041 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59_odh-ai-gateway-infra(400fd1c0-4622-455d-abe4-752fded45a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" podUID="400fd1c0-4622-455d-abe4-752fded45a70" Oct 30 19:45:03.980817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.980784 6514 generic.go:356] "Generic (PLEG): container finished" podID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerID="775d4a5ddf8be9d15b92499e54a1f9ccc997ab1c0c1a3b3bb628487a2447ab8d" exitCode=60 Oct 30 19:45:03.981340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.981311 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" event={"ID":"eaf20b59-5da1-4cc2-9480-c178b6c94d7b","Type":"ContainerDied","Data":"775d4a5ddf8be9d15b92499e54a1f9ccc997ab1c0c1a3b3bb628487a2447ab8d"} Oct 30 19:45:03.981514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.981496 6514 scope.go:122] "RemoveContainer" containerID="002477cc613b696e8dc073a9e66a3b9ef28cc47bf60d2942213698f61577ded2" Oct 30 19:45:03.982166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:03.982145 6514 scope.go:122] "RemoveContainer" containerID="775d4a5ddf8be9d15b92499e54a1f9ccc997ab1c0c1a3b3bb628487a2447ab8d" Oct 30 19:45:03.982730 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:03.982702 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f_odh-ai-gateway-infra(eaf20b59-5da1-4cc2-9480-c178b6c94d7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" Oct 30 19:45:04.166003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.165967 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" path="/var/lib/kubelet/pods/269a3f6e-538b-4be8-8a51-b056342c28fe/volumes" Oct 30 19:45:04.166909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.166888 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" path="/var/lib/kubelet/pods/e15ed13e-ac83-4499-b154-05c89333c29c/volumes" Oct 30 19:45:04.167661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.167640 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bf65bc-6c93-48e4-941f-a7ac20801141" path="/var/lib/kubelet/pods/e4bf65bc-6c93-48e4-941f-a7ac20801141/volumes" Oct 30 19:45:04.180402 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.180370 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" podUID="4d9968f0-efba-4537-8f96-359415fa147f" containerName="payload-processing" containerID="cri-o://f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc" gracePeriod=25 Oct 30 19:45:04.210039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.209988 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" podUID="a8463f9f-e220-4d14-835c-2ae558cb209a" containerName="payload-pre-processing" containerID="cri-o://e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6" gracePeriod=25 Oct 30 19:45:04.891939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.885103 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:45:05.029906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.951740 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume\") pod \"a8463f9f-e220-4d14-835c-2ae558cb209a\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " Oct 30 19:45:05.029906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:04.951791 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp\" (UniqueName: \"kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp\") pod \"a8463f9f-e220-4d14-835c-2ae558cb209a\" (UID: \"a8463f9f-e220-4d14-835c-2ae558cb209a\") " Oct 30 19:45:05.029906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.000077 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume" pod "a8463f9f-e220-4d14-835c-2ae558cb209a" (UID: "a8463f9f-e220-4d14-835c-2ae558cb209a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:05.029906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.001038 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8463f9f-e220-4d14-835c-2ae558cb209a-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.029906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.017746 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp" pod "a8463f9f-e220-4d14-835c-2ae558cb209a" (UID: "a8463f9f-e220-4d14-835c-2ae558cb209a"). InnerVolumeSpecName "kube-api-access-kgslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:05.081883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.038124 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl"] Oct 30 19:45:05.100981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.082113 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j"] Oct 30 19:45:05.121677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.120725 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:05.191103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.149792 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgslp\" (UniqueName: \"kubernetes.io/projected/a8463f9f-e220-4d14-835c-2ae558cb209a-kube-api-access-kgslp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.191103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.180545 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:45:05.219740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.213797 6514 generic.go:356] "Generic (PLEG): container finished" podID="4d9968f0-efba-4537-8f96-359415fa147f" containerID="f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc" exitCode=0 Oct 30 19:45:05.219740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.214012 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" event={"ID":"4d9968f0-efba-4537-8f96-359415fa147f","Type":"ContainerDied","Data":"f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc"} Oct 30 19:45:05.219740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.214071 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" event={"ID":"4d9968f0-efba-4537-8f96-359415fa147f","Type":"ContainerDied","Data":"63834c589c47018dc4ce3ff2e00d983cf6103ff9651d8e2dc23539a0a2fa9ed9"} Oct 30 19:45:05.219740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.214103 6514 scope.go:122] "RemoveContainer" containerID="f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc" Oct 30 19:45:05.232964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.232855 6514 generic.go:356] "Generic (PLEG): container finished" podID="a8463f9f-e220-4d14-835c-2ae558cb209a" containerID="e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6" exitCode=0 Oct 30 19:45:05.233621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.233568 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" Oct 30 19:45:05.236528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.233974 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" event={"ID":"a8463f9f-e220-4d14-835c-2ae558cb209a","Type":"ContainerDied","Data":"e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6"} Oct 30 19:45:05.236528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.234037 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4" event={"ID":"a8463f9f-e220-4d14-835c-2ae558cb209a","Type":"ContainerDied","Data":"86dd211b55089434b746875b45b395202a375749f91804f92aed83b3d9def4eb"} Oct 30 19:45:05.246531 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.246504 6514 scope.go:122] "RemoveContainer" containerID="775d4a5ddf8be9d15b92499e54a1f9ccc997ab1c0c1a3b3bb628487a2447ab8d" Oct 30 19:45:05.246991 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:05.246963 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f_odh-ai-gateway-infra(eaf20b59-5da1-4cc2-9480-c178b6c94d7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" Oct 30 19:45:05.252914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.252890 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume\") pod \"4d9968f0-efba-4537-8f96-359415fa147f\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " Oct 30 19:45:05.253281 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.253260 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v\" (UniqueName: \"kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v\") pod \"4d9968f0-efba-4537-8f96-359415fa147f\" (UID: \"4d9968f0-efba-4537-8f96-359415fa147f\") " Oct 30 19:45:05.257739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.257690 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v" pod "4d9968f0-efba-4537-8f96-359415fa147f" (UID: "4d9968f0-efba-4537-8f96-359415fa147f"). InnerVolumeSpecName "kube-api-access-8tw4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:05.258279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.258241 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume" pod "4d9968f0-efba-4537-8f96-359415fa147f" (UID: "4d9968f0-efba-4537-8f96-359415fa147f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:05.281635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.279159 6514 scope.go:122] "RemoveContainer" containerID="f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc" Oct 30 19:45:05.281635 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:05.280723 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc\": container with ID starting with f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc not found: ID does not exist" containerID="f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc" Oct 30 19:45:05.281635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.280765 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc"} err="failed to get container status \"f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc\": rpc error: code = NotFound desc = could not find container \"f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc\": container with ID starting with f7d296efa6b10a0e7d85b9e7db4bbab29c55b2e33c0fd009163ee03bc8f4a3fc not found: ID does not exist" Oct 30 19:45:05.281635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.280790 6514 scope.go:122] "RemoveContainer" containerID="e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6" Oct 30 19:45:05.297144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.297064 6514 scope.go:122] "RemoveContainer" containerID="e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6" Oct 30 19:45:05.297534 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:05.297484 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6\": container with ID starting with e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6 not found: ID does not exist" containerID="e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6" Oct 30 19:45:05.297534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.297510 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6"} err="failed to get container status \"e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6\": rpc error: code = NotFound desc = could not find container \"e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6\": container with ID starting with e9c00b3438df885ea7ab7ec7e4af3094c6e0a127f8a128d0202c48736df1eae6 not found: ID does not exist" Oct 30 19:45:05.320397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.320362 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4"] Oct 30 19:45:05.325093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.325066 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdb355-699984f479-clsd4"] Oct 30 19:45:05.341411 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.341258 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:05.356218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.356156 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tw4v\" (UniqueName: \"kubernetes.io/projected/4d9968f0-efba-4537-8f96-359415fa147f-kube-api-access-8tw4v\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.356218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.356185 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9968f0-efba-4537-8f96-359415fa147f-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.448423 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.448393 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:45:05.563884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.560353 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp\" (UniqueName: \"kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp\") pod \"8c930635-2467-4bc3-94fb-80f8bf3361c8\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " Oct 30 19:45:05.563884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.560830 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume\") pod \"8c930635-2467-4bc3-94fb-80f8bf3361c8\" (UID: \"8c930635-2467-4bc3-94fb-80f8bf3361c8\") " Oct 30 19:45:05.575515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.575096 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume" pod "8c930635-2467-4bc3-94fb-80f8bf3361c8" (UID: "8c930635-2467-4bc3-94fb-80f8bf3361c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:05.584116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.584075 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp" pod "8c930635-2467-4bc3-94fb-80f8bf3361c8" (UID: "8c930635-2467-4bc3-94fb-80f8bf3361c8"). InnerVolumeSpecName "kube-api-access-zvzpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:05.612904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.608332 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:45:05.663000 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.662969 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh\" (UniqueName: \"kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh\") pod \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " Oct 30 19:45:05.663213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.663197 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume\") pod \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\" (UID: \"4ed81f2a-e68a-42ab-b845-758d39b5fbae\") " Oct 30 19:45:05.663688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.663646 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume" pod "4ed81f2a-e68a-42ab-b845-758d39b5fbae" (UID: "4ed81f2a-e68a-42ab-b845-758d39b5fbae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:05.663791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.663729 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvzpp\" (UniqueName: \"kubernetes.io/projected/8c930635-2467-4bc3-94fb-80f8bf3361c8-kube-api-access-zvzpp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.663791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.663750 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c930635-2467-4bc3-94fb-80f8bf3361c8-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.665172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.665149 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh" pod "4ed81f2a-e68a-42ab-b845-758d39b5fbae" (UID: "4ed81f2a-e68a-42ab-b845-758d39b5fbae"). InnerVolumeSpecName "kube-api-access-mgjrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:05.765415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.765384 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjrh\" (UniqueName: \"kubernetes.io/projected/4ed81f2a-e68a-42ab-b845-758d39b5fbae-kube-api-access-mgjrh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.765415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:05.765411 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ed81f2a-e68a-42ab-b845-758d39b5fbae-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:06.158817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.158743 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8463f9f-e220-4d14-835c-2ae558cb209a" path="/var/lib/kubelet/pods/a8463f9f-e220-4d14-835c-2ae558cb209a/volumes" Oct 30 19:45:06.243920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.243886 6514 generic.go:356] "Generic (PLEG): container finished" podID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerID="616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3" exitCode=0 Oct 30 19:45:06.244067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.243936 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" Oct 30 19:45:06.244067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.243960 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" event={"ID":"4ed81f2a-e68a-42ab-b845-758d39b5fbae","Type":"ContainerDied","Data":"616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3"} Oct 30 19:45:06.244067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.244009 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9" event={"ID":"4ed81f2a-e68a-42ab-b845-758d39b5fbae","Type":"ContainerDied","Data":"987231154a8d048df8276b5a2facea513c5d082cc627abde0ba288fbc262caca"} Oct 30 19:45:06.244067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.244025 6514 scope.go:122] "RemoveContainer" containerID="616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3" Oct 30 19:45:06.245460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.245259 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc" Oct 30 19:45:06.246961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.246932 6514 generic.go:356] "Generic (PLEG): container finished" podID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerID="15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa" exitCode=0 Oct 30 19:45:06.247095 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.246981 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" Oct 30 19:45:06.247172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.247015 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" event={"ID":"8c930635-2467-4bc3-94fb-80f8bf3361c8","Type":"ContainerDied","Data":"15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa"} Oct 30 19:45:06.247172 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.247139 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv" event={"ID":"8c930635-2467-4bc3-94fb-80f8bf3361c8","Type":"ContainerDied","Data":"c4bd8194c79418021982c47af1c959ec567b59805ce14f187effa693812b3b53"} Oct 30 19:45:06.256458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.256439 6514 scope.go:122] "RemoveContainer" containerID="616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3" Oct 30 19:45:06.256733 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:06.256713 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3\": container with ID starting with 616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3 not found: ID does not exist" containerID="616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3" Oct 30 19:45:06.256810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.256742 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3"} err="failed to get container status \"616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3\": rpc error: code = NotFound desc = could not find container \"616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3\": container with ID starting with 616523a630e53f5ce8937610ebf6c707db4970a31def50fef0515c01339752e3 not found: ID does not exist" Oct 30 19:45:06.256810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.256758 6514 scope.go:122] "RemoveContainer" containerID="15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa" Oct 30 19:45:06.265908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.265885 6514 scope.go:122] "RemoveContainer" containerID="15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa" Oct 30 19:45:06.266211 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:06.266184 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa\": container with ID starting with 15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa not found: ID does not exist" containerID="15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa" Oct 30 19:45:06.266276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.266219 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa"} err="failed to get container status \"15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa\": rpc error: code = NotFound desc = could not find container \"15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa\": container with ID starting with 15b68ad7cffc26fb5ea57b1be6c4abd3a7d2509ac46e28e49a2821064457a9aa not found: ID does not exist" Oct 30 19:45:06.274696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.274670 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv"] Oct 30 19:45:06.277190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.277167 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d651c4-7f667dbdfb-48tkv"] Oct 30 19:45:06.289334 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.289309 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9"] Oct 30 19:45:06.291640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.291615 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d651c4-75cbc576bc-tp8q9"] Oct 30 19:45:06.304245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.304220 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc"] Oct 30 19:45:06.308811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.308789 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdb355-75d6685876-vw9vc"] Oct 30 19:45:06.503569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:06.503531 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:07.343543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:07.342590 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:08.159109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:08.159072 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9968f0-efba-4537-8f96-359415fa147f" path="/var/lib/kubelet/pods/4d9968f0-efba-4537-8f96-359415fa147f/volumes" Oct 30 19:45:08.159494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:08.159481 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" path="/var/lib/kubelet/pods/4ed81f2a-e68a-42ab-b845-758d39b5fbae/volumes" Oct 30 19:45:08.159826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:08.159815 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" path="/var/lib/kubelet/pods/8c930635-2467-4bc3-94fb-80f8bf3361c8/volumes" Oct 30 19:45:08.347887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:08.338486 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:08.656537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:08.656505 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:08.798763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:08.798715 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:09.012087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.012053 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9"] Oct 30 19:45:09.012902 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.012835 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" containerID="cri-o://42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503" gracePeriod=30 Oct 30 19:45:09.020188 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.020156 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q"] Oct 30 19:45:09.020395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.020372 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" containerID="cri-o://b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c" gracePeriod=30 Oct 30 19:45:09.030173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.030150 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs"] Oct 30 19:45:09.030323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.030306 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" containerID="cri-o://24315fe16050401907c852ca39f94802f4a7b2fd18b9872cf3297f81171fe6fe" gracePeriod=30 Oct 30 19:45:09.132890 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.132189 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:09.399091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.399002 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:09.717359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.717325 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:09.902393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:09.902357 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.039399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:10.039316 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerName="payload-processing" containerID="cri-o://0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74" gracePeriod=25 Oct 30 19:45:10.083174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:10.083136 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" containerName="payload-pre-processing" containerID="cri-o://9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467" gracePeriod=25 Oct 30 19:45:10.613507 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:10.613475 6514 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:10.613914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:10.613534 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:10.729985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:10.729942 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:11.803808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:11.803778 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.054827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.054746 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.149578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.149545 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd"] Oct 30 19:45:12.149796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.149774 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" containerID="cri-o://cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875" gracePeriod=30 Oct 30 19:45:12.286018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.285979 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.453766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.453722 6514 scope.go:122] "RemoveContainer" containerID="665929b2a5feff16b62cfab37f681fb1c69ba6d45fe06a456121f3ebab986783" Oct 30 19:45:12.522562 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.522511 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.685491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.685458 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.828711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.828633 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f"] Oct 30 19:45:12.901043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.901012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.991492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:12.991468 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:13.041899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.041451 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z\" (UniqueName: \"kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z\") pod \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " Oct 30 19:45:13.041899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.041538 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca\") pod \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\" (UID: \"eaf20b59-5da1-4cc2-9480-c178b6c94d7b\") " Oct 30 19:45:13.060891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.058203 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z" pod "eaf20b59-5da1-4cc2-9480-c178b6c94d7b" (UID: "eaf20b59-5da1-4cc2-9480-c178b6c94d7b"). InnerVolumeSpecName "kube-api-access-7262z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:13.066903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.066127 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca" pod "eaf20b59-5da1-4cc2-9480-c178b6c94d7b" (UID: "eaf20b59-5da1-4cc2-9480-c178b6c94d7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:13.066903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.066746 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:13.143810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.143736 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7262z\" (UniqueName: \"kubernetes.io/projected/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-kube-api-access-7262z\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:13.143810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.143763 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf20b59-5da1-4cc2-9480-c178b6c94d7b-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:13.349750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.349714 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" event={"ID":"eaf20b59-5da1-4cc2-9480-c178b6c94d7b","Type":"ContainerDied","Data":"eb01ffd13b1d35f107277ce00fcddc3f620b45748979942ed71ccb3ea75ad795"} Oct 30 19:45:13.349750 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.349743 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f" Oct 30 19:45:13.350028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.349786 6514 scope.go:122] "RemoveContainer" containerID="775d4a5ddf8be9d15b92499e54a1f9ccc997ab1c0c1a3b3bb628487a2447ab8d" Oct 30 19:45:13.387145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.387111 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:13.389190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.389169 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f"] Oct 30 19:45:13.436022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.435975 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3e35df-29889825-8s47f"] Oct 30 19:45:13.738351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.738265 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:13.974077 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:13.974046 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:14.152236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.152208 6514 scope.go:122] "RemoveContainer" containerID="070e0dc886620d474586ea62aee89e216b9430752687cafde0bb5fac125376ae" Oct 30 19:45:14.167291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.166847 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" path="/var/lib/kubelet/pods/eaf20b59-5da1-4cc2-9480-c178b6c94d7b/volumes" Oct 30 19:45:14.298414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.298379 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:45:14.329668 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.329382 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363412 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363456 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363544 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363595 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363622 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363665 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363715 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363760 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363801 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363829 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363882 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363939 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.363974 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.364006 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.364088 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.366588 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.364128 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.367642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.367609 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.368561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.368208 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.368561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.368366 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.368726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.368650 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.368855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.368830 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd\") pod \"bb4b2279-7bef-4b97-ab14-28417776470e\" (UID: \"bb4b2279-7bef-4b97-ab14-28417776470e\") " Oct 30 19:45:14.375002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.368941 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert\") pod \"45155a92-257c-47b0-a18e-38bff73f2120\" (UID: \"45155a92-257c-47b0-a18e-38bff73f2120\") " Oct 30 19:45:14.375002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.369617 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.375002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.369808 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.375002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.370087 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.375002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.374155 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "kube-api-access-xq5bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.375446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375136 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:14.375523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375502 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375526 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375541 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375554 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375570 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375586 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5bd\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-kube-api-access-xq5bd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375603 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45155a92-257c-47b0-a18e-38bff73f2120-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375619 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.375757 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375632 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.377016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.375924 6514 generic.go:356] "Generic (PLEG): container finished" podID="45155a92-257c-47b0-a18e-38bff73f2120" containerID="b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c" exitCode=0 Oct 30 19:45:14.377016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.376007 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" event={"ID":"45155a92-257c-47b0-a18e-38bff73f2120","Type":"ContainerDied","Data":"b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c"} Oct 30 19:45:14.377016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.376054 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" event={"ID":"45155a92-257c-47b0-a18e-38bff73f2120","Type":"ContainerDied","Data":"90508dc5a740d4d02a8beeddf0242bd337e0adfc2f10295ab4367a58509e0a4c"} Oct 30 19:45:14.377016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.376081 6514 scope.go:122] "RemoveContainer" containerID="b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c" Oct 30 19:45:14.377016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.376173 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q" Oct 30 19:45:14.377016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.376677 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.377779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.377203 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:14.381119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.381098 6514 generic.go:356] "Generic (PLEG): container finished" podID="984bd045-8081-46a7-a3a2-897b4210b861" containerID="24315fe16050401907c852ca39f94802f4a7b2fd18b9872cf3297f81171fe6fe" exitCode=0 Oct 30 19:45:14.381435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.381249 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" event={"ID":"984bd045-8081-46a7-a3a2-897b4210b861","Type":"ContainerDied","Data":"24315fe16050401907c852ca39f94802f4a7b2fd18b9872cf3297f81171fe6fe"} Oct 30 19:45:14.382982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.382641 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" containerID="cri-o://c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda" gracePeriod=45 Oct 30 19:45:14.384254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.384230 6514 generic.go:356] "Generic (PLEG): container finished" podID="bb4b2279-7bef-4b97-ab14-28417776470e" containerID="42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503" exitCode=0 Oct 30 19:45:14.384387 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.384369 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" event={"ID":"bb4b2279-7bef-4b97-ab14-28417776470e","Type":"ContainerDied","Data":"42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503"} Oct 30 19:45:14.384449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.384417 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" event={"ID":"bb4b2279-7bef-4b97-ab14-28417776470e","Type":"ContainerDied","Data":"8e4e71560b67a5137aa634a048178f553c6ca26b223f6d275757b4df8ae04110"} Oct 30 19:45:14.384511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.384494 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9" Oct 30 19:45:14.384756 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.384725 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "kube-api-access-pthbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.384966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.384944 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.385079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.385060 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:14.385417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.385392 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy" pod "bb4b2279-7bef-4b97-ab14-28417776470e" (UID: "bb4b2279-7bef-4b97-ab14-28417776470e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.385515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.385495 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.386247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.386201 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:14.389929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.389091 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy" pod "45155a92-257c-47b0-a18e-38bff73f2120" (UID: "45155a92-257c-47b0-a18e-38bff73f2120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.392911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.392761 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" containerID="cri-o://fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501" gracePeriod=45 Oct 30 19:45:14.397304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.396374 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": EOF" Oct 30 19:45:14.397304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.396990 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:45:14.397924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.397905 6514 scope.go:122] "RemoveContainer" containerID="b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c" Oct 30 19:45:14.399263 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:14.399235 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c\": container with ID starting with b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c not found: ID does not exist" containerID="b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c" Oct 30 19:45:14.399348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.399296 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c"} err="failed to get container status \"b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c\": rpc error: code = NotFound desc = could not find container \"b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c\": container with ID starting with b71c2e96a74d5f554e6dcadbf3a18af8b83f621dbd24947639ce6506ad73ef0c not found: ID does not exist" Oct 30 19:45:14.399348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.399321 6514 scope.go:122] "RemoveContainer" containerID="42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503" Oct 30 19:45:14.476498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476467 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476520 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476562 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476583 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476640 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476725 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476835 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476818 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476881 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.476927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.476892 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.477100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477002 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.477100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477006 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo\") pod \"984bd045-8081-46a7-a3a2-897b4210b861\" (UID: \"984bd045-8081-46a7-a3a2-897b4210b861\") " Oct 30 19:45:14.477100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477068 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.477100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477089 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:14.477348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477301 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.477688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477662 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477697 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477713 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477726 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bd045-8081-46a7-a3a2-897b4210b861-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477754 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477739 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb4b2279-7bef-4b97-ab14-28417776470e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477753 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45155a92-257c-47b0-a18e-38bff73f2120-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477766 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477782 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb4b2279-7bef-4b97-ab14-28417776470e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477796 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477811 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb4b2279-7bef-4b97-ab14-28417776470e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477824 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45155a92-257c-47b0-a18e-38bff73f2120-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477837 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477852 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2279-7bef-4b97-ab14-28417776470e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.477889 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthbd\" (UniqueName: \"kubernetes.io/projected/45155a92-257c-47b0-a18e-38bff73f2120-kube-api-access-pthbd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.478809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.478787 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.478851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.478819 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.479236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.479219 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:14.479309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.479256 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc" pod "984bd045-8081-46a7-a3a2-897b4210b861" (UID: "984bd045-8081-46a7-a3a2-897b4210b861"). InnerVolumeSpecName "kube-api-access-k4lhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.505416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.505395 6514 scope.go:122] "RemoveContainer" containerID="42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503" Oct 30 19:45:14.505776 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:14.505758 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503\": container with ID starting with 42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503 not found: ID does not exist" containerID="42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503" Oct 30 19:45:14.505837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.505785 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503"} err="failed to get container status \"42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503\": rpc error: code = NotFound desc = could not find container \"42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503\": container with ID starting with 42734eb8fc7be5e52f9c99385b67f66ae6d910762bb917196f396220647b2503 not found: ID does not exist" Oct 30 19:45:14.578788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.578754 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lhc\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-kube-api-access-k4lhc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.578788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.578786 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bd045-8081-46a7-a3a2-897b4210b861-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.578788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.578795 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bd045-8081-46a7-a3a2-897b4210b861-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.579093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.578804 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bd045-8081-46a7-a3a2-897b4210b861-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.644117 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.644082 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:45:14.677933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.677909 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:45:14.679166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679146 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home\") pod \"a7d8af65-c587-4d75-8161-b40d13e320a1\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " Oct 30 19:45:14.679259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679247 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf\" (UniqueName: \"kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf\") pod \"a7d8af65-c587-4d75-8161-b40d13e320a1\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " Oct 30 19:45:14.679303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679287 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache\") pod \"a7d8af65-c587-4d75-8161-b40d13e320a1\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " Oct 30 19:45:14.679347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679337 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir\") pod \"a7d8af65-c587-4d75-8161-b40d13e320a1\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " Oct 30 19:45:14.679428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679414 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs\") pod \"a7d8af65-c587-4d75-8161-b40d13e320a1\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " Oct 30 19:45:14.679495 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679420 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home" pod "a7d8af65-c587-4d75-8161-b40d13e320a1" (UID: "a7d8af65-c587-4d75-8161-b40d13e320a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.679561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679516 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm\") pod \"a7d8af65-c587-4d75-8161-b40d13e320a1\" (UID: \"a7d8af65-c587-4d75-8161-b40d13e320a1\") " Oct 30 19:45:14.679638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679601 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache" pod "a7d8af65-c587-4d75-8161-b40d13e320a1" (UID: "a7d8af65-c587-4d75-8161-b40d13e320a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.679726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.679690 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir" pod "a7d8af65-c587-4d75-8161-b40d13e320a1" (UID: "a7d8af65-c587-4d75-8161-b40d13e320a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.680146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.680124 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.680254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.680150 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.680254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.680167 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.681556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.681531 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs" pod "a7d8af65-c587-4d75-8161-b40d13e320a1" (UID: "a7d8af65-c587-4d75-8161-b40d13e320a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:14.681644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.681583 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm" pod "a7d8af65-c587-4d75-8161-b40d13e320a1" (UID: "a7d8af65-c587-4d75-8161-b40d13e320a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.681713 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.681697 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf" pod "a7d8af65-c587-4d75-8161-b40d13e320a1" (UID: "a7d8af65-c587-4d75-8161-b40d13e320a1"). InnerVolumeSpecName "kube-api-access-v7dhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.781432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.781388 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb\" (UniqueName: \"kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb\") pod \"9acb067e-d374-4017-95c6-761dcde158cb\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " Oct 30 19:45:14.781624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.781442 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm\") pod \"9acb067e-d374-4017-95c6-761dcde158cb\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " Oct 30 19:45:14.781624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.781511 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs\") pod \"9acb067e-d374-4017-95c6-761dcde158cb\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " Oct 30 19:45:14.781768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.781646 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir\") pod \"9acb067e-d374-4017-95c6-761dcde158cb\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " Oct 30 19:45:14.781768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.781690 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache\") pod \"9acb067e-d374-4017-95c6-761dcde158cb\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " Oct 30 19:45:14.781895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.781806 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home\") pod \"9acb067e-d374-4017-95c6-761dcde158cb\" (UID: \"9acb067e-d374-4017-95c6-761dcde158cb\") " Oct 30 19:45:14.782105 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782074 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache" pod "9acb067e-d374-4017-95c6-761dcde158cb" (UID: "9acb067e-d374-4017-95c6-761dcde158cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.782342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782318 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home" pod "9acb067e-d374-4017-95c6-761dcde158cb" (UID: "9acb067e-d374-4017-95c6-761dcde158cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.782509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782492 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.782573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782541 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d8af65-c587-4d75-8161-b40d13e320a1-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.782573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782556 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d8af65-c587-4d75-8161-b40d13e320a1-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.782573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782570 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.782734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782584 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7dhf\" (UniqueName: \"kubernetes.io/projected/a7d8af65-c587-4d75-8161-b40d13e320a1-kube-api-access-v7dhf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.782734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.782584 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir" pod "9acb067e-d374-4017-95c6-761dcde158cb" (UID: "9acb067e-d374-4017-95c6-761dcde158cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.783942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.783911 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm" pod "9acb067e-d374-4017-95c6-761dcde158cb" (UID: "9acb067e-d374-4017-95c6-761dcde158cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.784170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.784147 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb" pod "9acb067e-d374-4017-95c6-761dcde158cb" (UID: "9acb067e-d374-4017-95c6-761dcde158cb"). InnerVolumeSpecName "kube-api-access-vrfvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.784987 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.784961 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs" pod "9acb067e-d374-4017-95c6-761dcde158cb" (UID: "9acb067e-d374-4017-95c6-761dcde158cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:14.858807 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.858724 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9"] Oct 30 19:45:14.865841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.865810 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-d7zl9"] Oct 30 19:45:14.883572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.883542 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.883572 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.883571 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfvb\" (UniqueName: \"kubernetes.io/projected/9acb067e-d374-4017-95c6-761dcde158cb-kube-api-access-vrfvb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.883738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.883586 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9acb067e-d374-4017-95c6-761dcde158cb-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.883738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.883603 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acb067e-d374-4017-95c6-761dcde158cb-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.896331 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.896302 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q"] Oct 30 19:45:14.910563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:14.910536 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-57r8q"] Oct 30 19:45:15.031809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.031771 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:15.392804 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.392765 6514 generic.go:356] "Generic (PLEG): container finished" podID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerID="c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda" exitCode=0 Oct 30 19:45:15.392971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.392820 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" Oct 30 19:45:15.392971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.392851 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" event={"ID":"a7d8af65-c587-4d75-8161-b40d13e320a1","Type":"ContainerDied","Data":"c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda"} Oct 30 19:45:15.392971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.392911 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt" event={"ID":"a7d8af65-c587-4d75-8161-b40d13e320a1","Type":"ContainerDied","Data":"56d8cc13a413e81ea0d8c080458c1b448039d06f93cba07b85d88f1ba5e32485"} Oct 30 19:45:15.392971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.392935 6514 scope.go:122] "RemoveContainer" containerID="c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda" Oct 30 19:45:15.395586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.395564 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" Oct 30 19:45:15.395725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.395563 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs" event={"ID":"984bd045-8081-46a7-a3a2-897b4210b861","Type":"ContainerDied","Data":"dd9a2dbe3914e29facf7bb96987267924f453311b2fa3c0d06a5fdf0dfbd45a9"} Oct 30 19:45:15.397783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.397759 6514 generic.go:356] "Generic (PLEG): container finished" podID="400fd1c0-4622-455d-abe4-752fded45a70" containerID="4c96642c3445f7766295eae9b3ad4611c92891c67b9b5a338d366cc62f25e39c" exitCode=60 Oct 30 19:45:15.397935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.397834 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" event={"ID":"400fd1c0-4622-455d-abe4-752fded45a70","Type":"ContainerDied","Data":"4c96642c3445f7766295eae9b3ad4611c92891c67b9b5a338d366cc62f25e39c"} Oct 30 19:45:15.398268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.398247 6514 scope.go:122] "RemoveContainer" containerID="4c96642c3445f7766295eae9b3ad4611c92891c67b9b5a338d366cc62f25e39c" Oct 30 19:45:15.398544 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:15.398522 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59_odh-ai-gateway-infra(400fd1c0-4622-455d-abe4-752fded45a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" podUID="400fd1c0-4622-455d-abe4-752fded45a70" Oct 30 19:45:15.400914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.400888 6514 generic.go:356] "Generic (PLEG): container finished" podID="9acb067e-d374-4017-95c6-761dcde158cb" containerID="fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501" exitCode=0 Oct 30 19:45:15.401040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.400951 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" event={"ID":"9acb067e-d374-4017-95c6-761dcde158cb","Type":"ContainerDied","Data":"fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501"} Oct 30 19:45:15.401040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.400963 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" Oct 30 19:45:15.401040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.400981 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc" event={"ID":"9acb067e-d374-4017-95c6-761dcde158cb","Type":"ContainerDied","Data":"27a724a04e5440b9b9382b5f0486235c37072a0e92254ae39c50ac89e02ba236"} Oct 30 19:45:15.406072 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.406053 6514 scope.go:122] "RemoveContainer" containerID="c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda" Oct 30 19:45:15.411169 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:15.411142 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda\": container with ID starting with c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda not found: ID does not exist" containerID="c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda" Oct 30 19:45:15.411282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.411180 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda"} err="failed to get container status \"c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda\": rpc error: code = NotFound desc = could not find container \"c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda\": container with ID starting with c6c3a148f9bf44371ed368a9cd5062e59f7374ce3f8fb50ff912227b855ddeda not found: ID does not exist" Oct 30 19:45:15.411282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.411203 6514 scope.go:122] "RemoveContainer" containerID="24315fe16050401907c852ca39f94802f4a7b2fd18b9872cf3297f81171fe6fe" Oct 30 19:45:15.425748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.425726 6514 scope.go:122] "RemoveContainer" containerID="070e0dc886620d474586ea62aee89e216b9430752687cafde0bb5fac125376ae" Oct 30 19:45:15.441808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.441784 6514 scope.go:122] "RemoveContainer" containerID="fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501" Oct 30 19:45:15.452041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.452010 6514 scope.go:122] "RemoveContainer" containerID="fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501" Oct 30 19:45:15.452332 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:15.452310 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501\": container with ID starting with fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501 not found: ID does not exist" containerID="fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501" Oct 30 19:45:15.452397 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.452346 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501"} err="failed to get container status \"fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501\": rpc error: code = NotFound desc = could not find container \"fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501\": container with ID starting with fa3866fc7616d8853b99b89dc98571cc4f3618344b3ab8c811cedd21b6d38501 not found: ID does not exist" Oct 30 19:45:15.466921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.466899 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc"] Oct 30 19:45:15.472615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.472593 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/simulator-w1-419d0a-kserve-7ff5877979-ts7nc"] Oct 30 19:45:15.492277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.492253 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs"] Oct 30 19:45:15.497662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.497638 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-9z5vs"] Oct 30 19:45:15.511654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.511630 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt"] Oct 30 19:45:15.516920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:15.516899 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/premium-w1-419d0a-kserve-6dbd9d446b-5spqt"] Oct 30 19:45:16.153772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.153737 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:16.158563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.158534 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45155a92-257c-47b0-a18e-38bff73f2120" path="/var/lib/kubelet/pods/45155a92-257c-47b0-a18e-38bff73f2120/volumes" Oct 30 19:45:16.159132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.159113 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984bd045-8081-46a7-a3a2-897b4210b861" path="/var/lib/kubelet/pods/984bd045-8081-46a7-a3a2-897b4210b861/volumes" Oct 30 19:45:16.159559 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.159540 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acb067e-d374-4017-95c6-761dcde158cb" path="/var/lib/kubelet/pods/9acb067e-d374-4017-95c6-761dcde158cb/volumes" Oct 30 19:45:16.160235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.160211 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" path="/var/lib/kubelet/pods/a7d8af65-c587-4d75-8161-b40d13e320a1/volumes" Oct 30 19:45:16.160628 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.160615 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" path="/var/lib/kubelet/pods/bb4b2279-7bef-4b97-ab14-28417776470e/volumes" Oct 30 19:45:16.338458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.338426 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:16.338605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.338426 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:16.459523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.459489 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59"] Oct 30 19:45:16.615155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.615119 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:16.703632 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.703605 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca\") pod \"400fd1c0-4622-455d-abe4-752fded45a70\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " Oct 30 19:45:16.703813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.703798 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv\" (UniqueName: \"kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv\") pod \"400fd1c0-4622-455d-abe4-752fded45a70\" (UID: \"400fd1c0-4622-455d-abe4-752fded45a70\") " Oct 30 19:45:16.703990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.703969 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca" pod "400fd1c0-4622-455d-abe4-752fded45a70" (UID: "400fd1c0-4622-455d-abe4-752fded45a70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:16.704266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.704242 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/400fd1c0-4622-455d-abe4-752fded45a70-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.705920 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.705899 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv" pod "400fd1c0-4622-455d-abe4-752fded45a70" (UID: "400fd1c0-4622-455d-abe4-752fded45a70"). InnerVolumeSpecName "kube-api-access-mswlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.805662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:16.805633 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mswlv\" (UniqueName: \"kubernetes.io/projected/400fd1c0-4622-455d-abe4-752fded45a70-kube-api-access-mswlv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.228149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.228119 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:17.413460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.413432 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:45:17.417165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.417144 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" Oct 30 19:45:17.417309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.417178 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59" event={"ID":"400fd1c0-4622-455d-abe4-752fded45a70","Type":"ContainerDied","Data":"a319d70e381b7a0c1b8a4b8fce6eb13b2f4948e8b915282db8df93cbf2a2f75d"} Oct 30 19:45:17.417309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.417250 6514 scope.go:122] "RemoveContainer" containerID="4c96642c3445f7766295eae9b3ad4611c92891c67b9b5a338d366cc62f25e39c" Oct 30 19:45:17.419171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.419135 6514 generic.go:356] "Generic (PLEG): container finished" podID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerID="cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875" exitCode=0 Oct 30 19:45:17.419278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.419241 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" event={"ID":"78eb4588-c43b-4925-99c5-73c0183e10ba","Type":"ContainerDied","Data":"cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875"} Oct 30 19:45:17.419343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.419286 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" event={"ID":"78eb4588-c43b-4925-99c5-73c0183e10ba","Type":"ContainerDied","Data":"c53917f3b0da1386971edab155eee287d754690e69445c799ef85a3aa8adbb96"} Oct 30 19:45:17.419601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.419376 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd" Oct 30 19:45:17.427089 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.427064 6514 scope.go:122] "RemoveContainer" containerID="cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875" Oct 30 19:45:17.436127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.436110 6514 scope.go:122] "RemoveContainer" containerID="a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9" Oct 30 19:45:17.445019 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.445000 6514 scope.go:122] "RemoveContainer" containerID="cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875" Oct 30 19:45:17.445321 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:17.445301 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875\": container with ID starting with cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875 not found: ID does not exist" containerID="cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875" Oct 30 19:45:17.445377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.445331 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875"} err="failed to get container status \"cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875\": rpc error: code = NotFound desc = could not find container \"cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875\": container with ID starting with cd118cb58436c47b856b994ee3494c7080ee244a325ffb436be89bc613b97875 not found: ID does not exist" Oct 30 19:45:17.445377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.445349 6514 scope.go:122] "RemoveContainer" containerID="a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9" Oct 30 19:45:17.445602 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:17.445581 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9\": container with ID starting with a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9 not found: ID does not exist" containerID="a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9" Oct 30 19:45:17.445684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.445611 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9"} err="failed to get container status \"a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9\": rpc error: code = NotFound desc = could not find container \"a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9\": container with ID starting with a88ed9fae86bfcbdcacb3bbb339683661597c29f7bf50d56c142adf9e43d92d9 not found: ID does not exist" Oct 30 19:45:17.510679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.510641 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59"] Oct 30 19:45:17.515324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.515293 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-47110f-29889825-rxk59"] Oct 30 19:45:17.517728 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517702 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.517826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517747 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.517826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517772 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.517826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517805 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.518055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517841 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.518055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517947 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.518055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.517984 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.518055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518042 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.518269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518072 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.518269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518101 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.518269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518112 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data\") pod \"78eb4588-c43b-4925-99c5-73c0183e10ba\" (UID: \"78eb4588-c43b-4925-99c5-73c0183e10ba\") " Oct 30 19:45:17.518426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518281 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:17.518426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518302 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.518532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518478 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.518729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518706 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.518799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518734 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.518799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518748 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.518799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518762 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb4588-c43b-4925-99c5-73c0183e10ba-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.518799 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.518777 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.520941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.520918 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:17.521013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.520976 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.521213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.521195 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:17.521382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.521366 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst" pod "78eb4588-c43b-4925-99c5-73c0183e10ba" (UID: "78eb4588-c43b-4925-99c5-73c0183e10ba"). InnerVolumeSpecName "kube-api-access-vpxst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.619681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.619649 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.619681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.619675 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.619844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.619685 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxst\" (UniqueName: \"kubernetes.io/projected/78eb4588-c43b-4925-99c5-73c0183e10ba-kube-api-access-vpxst\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.619844 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.619696 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb4588-c43b-4925-99c5-73c0183e10ba-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.754491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.754456 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd"] Oct 30 19:45:17.766014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:17.765983 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-fdb355-openshift-default-d67d8d799-b6tcd"] Oct 30 19:45:18.158590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:18.158517 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400fd1c0-4622-455d-abe4-752fded45a70" path="/var/lib/kubelet/pods/400fd1c0-4622-455d-abe4-752fded45a70/volumes" Oct 30 19:45:18.159068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:18.159047 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" path="/var/lib/kubelet/pods/78eb4588-c43b-4925-99c5-73c0183e10ba/volumes" Oct 30 19:45:18.358319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:18.358287 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:19.422959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:19.416712 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:20.423905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:20.423440 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:20.881963 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:20.881921 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:45:21.158839 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:21.158753 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:45:21.262735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:21.262700 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:22.309503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:22.309469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:22.530016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:22.529962 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:22.912244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:22.912210 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n"] Oct 30 19:45:23.034271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.034239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.149174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.149142 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs"] Oct 30 19:45:23.149466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.149440 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" podUID="93e751e3-0511-411a-a275-cb9359879170" containerName="maas-api" containerID="cri-o://4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709" gracePeriod=30 Oct 30 19:45:23.212960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.212927 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.442647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.438806 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.468148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.468110 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:45:23.490109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.490074 6514 generic.go:356] "Generic (PLEG): container finished" podID="93e751e3-0511-411a-a275-cb9359879170" containerID="4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709" exitCode=0 Oct 30 19:45:23.490541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.490149 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" Oct 30 19:45:23.490541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.490265 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" event={"ID":"93e751e3-0511-411a-a275-cb9359879170","Type":"ContainerDied","Data":"4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709"} Oct 30 19:45:23.490541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.490320 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs" event={"ID":"93e751e3-0511-411a-a275-cb9359879170","Type":"ContainerDied","Data":"4ffa18997ca6af5aa24f156ff8317920e80f43f1f135b125d486896b9a494a46"} Oct 30 19:45:23.490541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.490355 6514 scope.go:122] "RemoveContainer" containerID="4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709" Oct 30 19:45:23.507290 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.507271 6514 scope.go:122] "RemoveContainer" containerID="4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709" Oct 30 19:45:23.507803 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:23.507782 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709\": container with ID starting with 4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709 not found: ID does not exist" containerID="4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709" Oct 30 19:45:23.507803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.507807 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709"} err="failed to get container status \"4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709\": rpc error: code = NotFound desc = could not find container \"4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709\": container with ID starting with 4074af4bc208d3466651607d736104cbb5c12a223e4a24fdfefc910859cdd709 not found: ID does not exist" Oct 30 19:45:23.633002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.632965 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq\" (UniqueName: \"kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq\") pod \"93e751e3-0511-411a-a275-cb9359879170\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " Oct 30 19:45:23.633002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.633007 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca\") pod \"93e751e3-0511-411a-a275-cb9359879170\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " Oct 30 19:45:23.633233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.633028 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\") pod \"93e751e3-0511-411a-a275-cb9359879170\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " Oct 30 19:45:23.633310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.633290 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs\") pod \"93e751e3-0511-411a-a275-cb9359879170\" (UID: \"93e751e3-0511-411a-a275-cb9359879170\") " Oct 30 19:45:23.633427 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.633402 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca" pod "93e751e3-0511-411a-a275-cb9359879170" (UID: "93e751e3-0511-411a-a275-cb9359879170"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:23.633772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.633747 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e751e3-0511-411a-a275-cb9359879170-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.635316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.635291 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq" pod "93e751e3-0511-411a-a275-cb9359879170" (UID: "93e751e3-0511-411a-a275-cb9359879170"). InnerVolumeSpecName "kube-api-access-5frxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:23.635445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.635294 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls" pod "93e751e3-0511-411a-a275-cb9359879170" (UID: "93e751e3-0511-411a-a275-cb9359879170"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:23.635445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.635416 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs" pod "93e751e3-0511-411a-a275-cb9359879170" (UID: "93e751e3-0511-411a-a275-cb9359879170"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:23.664925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.661012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.735058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.734987 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.735058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.735016 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5frxq\" (UniqueName: \"kubernetes.io/projected/93e751e3-0511-411a-a275-cb9359879170-kube-api-access-5frxq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.735058 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.735029 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e751e3-0511-411a-a275-cb9359879170-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.822736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.822707 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs"] Oct 30 19:45:23.826465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:23.826441 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-976b80-7795488bf4-nswcs"] Oct 30 19:45:24.158759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:24.158676 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e751e3-0511-411a-a275-cb9359879170" path="/var/lib/kubelet/pods/93e751e3-0511-411a-a275-cb9359879170/volumes" Oct 30 19:45:24.188209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:24.188182 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:24.366503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:24.366470 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:24.627767 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:24.627732 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:24.874667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:24.874638 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.079915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:25.079779 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.258075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:25.258033 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.656931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:25.656903 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:26.112528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:26.112471 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:26.338967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:26.338931 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:26.339125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:26.338977 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:26.988369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:26.988327 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.911950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:27.911916 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.912731 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:27.912709 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" podUID="2552fab8-b507-4c24-b3a4-fd1e06c51835" containerName="payload-pre-processing" containerID="cri-o://39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912" gracePeriod=25 Oct 30 19:45:28.076042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.076003 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.182340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.182314 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:45:28.312139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.312107 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475\" (UniqueName: \"kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475\") pod \"2552fab8-b507-4c24-b3a4-fd1e06c51835\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " Oct 30 19:45:28.312308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.312270 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume\") pod \"2552fab8-b507-4c24-b3a4-fd1e06c51835\" (UID: \"2552fab8-b507-4c24-b3a4-fd1e06c51835\") " Oct 30 19:45:28.312742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.312706 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume" pod "2552fab8-b507-4c24-b3a4-fd1e06c51835" (UID: "2552fab8-b507-4c24-b3a4-fd1e06c51835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:28.312908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.312843 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2552fab8-b507-4c24-b3a4-fd1e06c51835-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:28.314182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.314161 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475" pod "2552fab8-b507-4c24-b3a4-fd1e06c51835" (UID: "2552fab8-b507-4c24-b3a4-fd1e06c51835"). InnerVolumeSpecName "kube-api-access-mt475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:28.413706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.413639 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt475\" (UniqueName: \"kubernetes.io/projected/2552fab8-b507-4c24-b3a4-fd1e06c51835-kube-api-access-mt475\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:28.507648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.507618 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.523145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.523117 6514 generic.go:356] "Generic (PLEG): container finished" podID="2552fab8-b507-4c24-b3a4-fd1e06c51835" containerID="39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912" exitCode=0 Oct 30 19:45:28.523279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.523172 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" Oct 30 19:45:28.523279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.523212 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" event={"ID":"2552fab8-b507-4c24-b3a4-fd1e06c51835","Type":"ContainerDied","Data":"39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912"} Oct 30 19:45:28.523279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.523264 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n" event={"ID":"2552fab8-b507-4c24-b3a4-fd1e06c51835","Type":"ContainerDied","Data":"424851b18c25eb09e0c3e6694791288b5c82b5b98f1bcdc88eff2f92fec6b281"} Oct 30 19:45:28.523376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.523286 6514 scope.go:122] "RemoveContainer" containerID="39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912" Oct 30 19:45:28.532674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.532658 6514 scope.go:122] "RemoveContainer" containerID="39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912" Oct 30 19:45:28.532918 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:28.532899 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912\": container with ID starting with 39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912 not found: ID does not exist" containerID="39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912" Oct 30 19:45:28.532985 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.532925 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912"} err="failed to get container status \"39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912\": rpc error: code = NotFound desc = could not find container \"39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912\": container with ID starting with 39f7de856f097cf435a5c1e3c952d938f5232c048145768a564320de86d0e912 not found: ID does not exist" Oct 30 19:45:28.551096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.551068 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n"] Oct 30 19:45:28.555087 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.555060 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-976b80-6869f686f9-9cs2n"] Oct 30 19:45:28.957527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:28.957490 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:29.775193 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:29.775160 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:30.158506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.158478 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2552fab8-b507-4c24-b3a4-fd1e06c51835" path="/var/lib/kubelet/pods/2552fab8-b507-4c24-b3a4-fd1e06c51835/volumes" Oct 30 19:45:30.350324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.350300 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:45:30.374185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.374161 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:45:30.435662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.435631 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume\") pod \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " Oct 30 19:45:30.435833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.435750 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume\") pod \"e19b0962-4976-45a6-bffa-06720cab3b25\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " Oct 30 19:45:30.435903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.435829 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q\" (UniqueName: \"kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q\") pod \"e19b0962-4976-45a6-bffa-06720cab3b25\" (UID: \"e19b0962-4976-45a6-bffa-06720cab3b25\") " Oct 30 19:45:30.435980 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.435962 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs\" (UniqueName: \"kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs\") pod \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\" (UID: \"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe\") " Oct 30 19:45:30.436209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.436184 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume" pod "0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" (UID: "0d975d56-a1b1-4cdf-9dd1-3691ccabafbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:30.436301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.436246 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume" pod "e19b0962-4976-45a6-bffa-06720cab3b25" (UID: "e19b0962-4976-45a6-bffa-06720cab3b25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:30.436563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.436535 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19b0962-4976-45a6-bffa-06720cab3b25-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.436684 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.436566 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.438114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.438092 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs" pod "0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" (UID: "0d975d56-a1b1-4cdf-9dd1-3691ccabafbe"). InnerVolumeSpecName "kube-api-access-k8vcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:30.438216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.438156 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q" pod "e19b0962-4976-45a6-bffa-06720cab3b25" (UID: "e19b0962-4976-45a6-bffa-06720cab3b25"). InnerVolumeSpecName "kube-api-access-77t8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:30.537376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.537346 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77t8q\" (UniqueName: \"kubernetes.io/projected/e19b0962-4976-45a6-bffa-06720cab3b25-kube-api-access-77t8q\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.537376 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.537377 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vcs\" (UniqueName: \"kubernetes.io/projected/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe-kube-api-access-k8vcs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.537849 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.537824 6514 generic.go:356] "Generic (PLEG): container finished" podID="e19b0962-4976-45a6-bffa-06720cab3b25" containerID="9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467" exitCode=0 Oct 30 19:45:30.537940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.537905 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" Oct 30 19:45:30.537940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.537910 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" event={"ID":"e19b0962-4976-45a6-bffa-06720cab3b25","Type":"ContainerDied","Data":"9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467"} Oct 30 19:45:30.538041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.537976 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j" event={"ID":"e19b0962-4976-45a6-bffa-06720cab3b25","Type":"ContainerDied","Data":"688181f22f1d6f0e393ea718fe177927b3c2f1073cce8dca5eee20576bb75085"} Oct 30 19:45:30.538041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.538011 6514 scope.go:122] "RemoveContainer" containerID="9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467" Oct 30 19:45:30.539594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.539567 6514 generic.go:356] "Generic (PLEG): container finished" podID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerID="0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74" exitCode=0 Oct 30 19:45:30.539718 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.539602 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" event={"ID":"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe","Type":"ContainerDied","Data":"0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74"} Oct 30 19:45:30.539718 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.539627 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" Oct 30 19:45:30.539718 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.539633 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl" event={"ID":"0d975d56-a1b1-4cdf-9dd1-3691ccabafbe","Type":"ContainerDied","Data":"e06e87b1f4cea7a8e2de22eab2c8992bcfa435e9c86fe8e459771d1def45329b"} Oct 30 19:45:30.548899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.548853 6514 scope.go:122] "RemoveContainer" containerID="9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467" Oct 30 19:45:30.549180 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:30.549158 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467\": container with ID starting with 9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467 not found: ID does not exist" containerID="9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467" Oct 30 19:45:30.549275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.549187 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467"} err="failed to get container status \"9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467\": rpc error: code = NotFound desc = could not find container \"9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467\": container with ID starting with 9451d95656bf841679d4e7c6c04bddebb9a20cd81748b4d45c960adfc8e17467 not found: ID does not exist" Oct 30 19:45:30.549275 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.549206 6514 scope.go:122] "RemoveContainer" containerID="0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74" Oct 30 19:45:30.557638 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.557621 6514 scope.go:122] "RemoveContainer" containerID="0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74" Oct 30 19:45:30.557937 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:30.557912 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74\": container with ID starting with 0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74 not found: ID does not exist" containerID="0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74" Oct 30 19:45:30.558035 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.557945 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74"} err="failed to get container status \"0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74\": rpc error: code = NotFound desc = could not find container \"0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74\": container with ID starting with 0817f7eeaeca293d60786fb6879a5a0a39f86ac893131667f816f9386adf8d74 not found: ID does not exist" Oct 30 19:45:30.564573 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.564553 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j"] Oct 30 19:45:30.569084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.569066 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-419d0a-57688785ff-fjn4j"] Oct 30 19:45:30.592735 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.592706 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl"] Oct 30 19:45:30.644444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.644401 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-419d0a-688c6dfd6-8qmrl"] Oct 30 19:45:30.924944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:30.920685 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.036794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:32.036760 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.157850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:32.157818 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" path="/var/lib/kubelet/pods/0d975d56-a1b1-4cdf-9dd1-3691ccabafbe/volumes" Oct 30 19:45:32.158267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:32.158252 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" path="/var/lib/kubelet/pods/e19b0962-4976-45a6-bffa-06720cab3b25/volumes" Oct 30 19:45:32.892823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:32.892787 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:33.958149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:33.958120 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:34.967555 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:34.967472 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:35.200484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.200447 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:35.293003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.292914 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn"] Oct 30 19:45:35.293286 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.293127 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" containerID="cri-o://e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2" gracePeriod=30 Oct 30 19:45:35.307148 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.307121 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9"] Oct 30 19:45:35.307305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.307287 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" containerID="cri-o://0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9" gracePeriod=30 Oct 30 19:45:35.402544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.402506 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:35.547527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.547446 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:35.749404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:35.749370 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:36.131626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:36.131583 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:36.884676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:36.884636 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:45:37.203241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:37.203203 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:37.533207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:37.533127 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:37.705342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:37.705299 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.166164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.166131 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.330886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.327826 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.509993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.503341 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.724717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.724681 6514 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:38.724913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.724768 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:38.769657 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.769574 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.976102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:38.976072 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.745360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:39.745328 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.972643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:39.972609 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:40.178015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.177926 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:40.565872 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.565843 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:45:40.604819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.604796 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:45:40.605346 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.605323 6514 generic.go:356] "Generic (PLEG): container finished" podID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerID="e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2" exitCode=0 Oct 30 19:45:40.605445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.605371 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" Oct 30 19:45:40.605445 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.605399 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" event={"ID":"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab","Type":"ContainerDied","Data":"e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2"} Oct 30 19:45:40.605557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.605448 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn" event={"ID":"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab","Type":"ContainerDied","Data":"3c17f3847de2578fd1f8af16e6167ca1d52f2387f80e629374f6f5eb524ed493"} Oct 30 19:45:40.605557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.605467 6514 scope.go:122] "RemoveContainer" containerID="e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2" Oct 30 19:45:40.606733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.606709 6514 generic.go:356] "Generic (PLEG): container finished" podID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerID="0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9" exitCode=0 Oct 30 19:45:40.606841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.606746 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" Oct 30 19:45:40.606841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.606793 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" event={"ID":"5b58f0f0-0f6a-4189-92c0-0921200c5c3d","Type":"ContainerDied","Data":"0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9"} Oct 30 19:45:40.606841 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.606834 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9" event={"ID":"5b58f0f0-0f6a-4189-92c0-0921200c5c3d","Type":"ContainerDied","Data":"33715a9b3600750fe2c9ef1779ae755390483f87abe3852a92e197d8f7a629f7"} Oct 30 19:45:40.617408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.617391 6514 scope.go:122] "RemoveContainer" containerID="e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2" Oct 30 19:45:40.617708 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:40.617682 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2\": container with ID starting with e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2 not found: ID does not exist" containerID="e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2" Oct 30 19:45:40.617753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.617720 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2"} err="failed to get container status \"e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2\": rpc error: code = NotFound desc = could not find container \"e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2\": container with ID starting with e99c66f6ea678bfcffd82cdbf5b84423ade7814befe9a3f3014ec8fd94d2f5d2 not found: ID does not exist" Oct 30 19:45:40.617753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.617736 6514 scope.go:122] "RemoveContainer" containerID="0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9" Oct 30 19:45:40.625892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.625858 6514 scope.go:122] "RemoveContainer" containerID="0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9" Oct 30 19:45:40.626154 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:40.626136 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9\": container with ID starting with 0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9 not found: ID does not exist" containerID="0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9" Oct 30 19:45:40.626212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.626161 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9"} err="failed to get container status \"0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9\": rpc error: code = NotFound desc = could not find container \"0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9\": container with ID starting with 0c237432a284f0330093c95800f4ba7828418610c494f02fd00f165a559062d9 not found: ID does not exist" Oct 30 19:45:40.673084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673056 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673098 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673129 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673201 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673243 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673284 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673327 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673366 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673423 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673495 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673535 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673586 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673649 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673676 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673723 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673793 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.673961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673823 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket\") pod \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\" (UID: \"8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab\") " Oct 30 19:45:40.673961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673832 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:40.673961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.673857 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r\") pod \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\" (UID: \"5b58f0f0-0f6a-4189-92c0-0921200c5c3d\") " Oct 30 19:45:40.674621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.674393 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.674734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.674646 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.674801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.674785 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.674877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.674823 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.675254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.675014 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.675974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.675412 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.675974 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.675672 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.676701 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.676662 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:40.676945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.676923 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.677196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.677174 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.679080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.679049 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "kube-api-access-qjz5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:40.679257 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.679225 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "kube-api-access-rp9hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:40.679374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.679327 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:40.679374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.679327 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:40.679653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.679633 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.679653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.679638 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:40.680122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.680093 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy" pod "5b58f0f0-0f6a-4189-92c0-0921200c5c3d" (UID: "5b58f0f0-0f6a-4189-92c0-0921200c5c3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.681031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.681008 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo" pod "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" (UID: "8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775506 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775531 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775541 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9hz\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-kube-api-access-rp9hz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775550 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775559 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775572 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.775582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775581 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775589 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775597 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775606 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775615 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775623 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775632 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775640 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775648 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775656 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.776222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.775664 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjz5r\" (UniqueName: \"kubernetes.io/projected/5b58f0f0-0f6a-4189-92c0-0921200c5c3d-kube-api-access-qjz5r\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.965075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.965039 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9"] Oct 30 19:45:40.973261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:40.973230 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6snr9"] Oct 30 19:45:41.014173 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:41.014141 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn"] Oct 30 19:45:41.026606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:41.026538 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-6x6nn"] Oct 30 19:45:41.382646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:41.382614 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:42.158266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:42.158230 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" path="/var/lib/kubelet/pods/5b58f0f0-0f6a-4189-92c0-0921200c5c3d/volumes" Oct 30 19:45:42.158712 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:42.158698 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" path="/var/lib/kubelet/pods/8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab/volumes" Oct 30 19:45:42.371461 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:42.371392 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:43.109021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:43.108989 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:43.357992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:43.357950 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.048719 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:44.048675 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.408196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:44.408094 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:45.278356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:45.278297 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:46.342584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:46.342550 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:46.510914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:46.510848 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n"] Oct 30 19:45:46.511128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:46.511094 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" containerID="cri-o://1c3ba15c970e28e9b1b44493207109256d46d1a496784b86bf66f70401d5a2d0" gracePeriod=30 Oct 30 19:45:47.271539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:47.271503 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:47.459478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:47.459445 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:47.691637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:47.691605 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:48.234163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:48.234129 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:48.272533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:48.272505 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:45:48.435117 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:48.435006 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:49.269451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:49.269418 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw"] Oct 30 19:45:49.296967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:49.296932 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc"] Oct 30 19:45:50.277430 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:50.277396 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.773415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:50.773367 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.881476 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:45:50.881435 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:45:51.636949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.636850 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.679680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.679633 6514 generic.go:356] "Generic (PLEG): container finished" podID="839f94be-9343-4fc5-830b-d489bbdb4059" containerID="1c3ba15c970e28e9b1b44493207109256d46d1a496784b86bf66f70401d5a2d0" exitCode=0 Oct 30 19:45:51.679837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.679701 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" event={"ID":"839f94be-9343-4fc5-830b-d489bbdb4059","Type":"ContainerDied","Data":"1c3ba15c970e28e9b1b44493207109256d46d1a496784b86bf66f70401d5a2d0"} Oct 30 19:45:51.778676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.778615 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:45:51.809080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.809041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.912993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.912964 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913006 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913045 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913105 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913175 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913252 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913316 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913344 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913416 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913394 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:51.913621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913401 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:51.913621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913513 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert\") pod \"839f94be-9343-4fc5-830b-d489bbdb4059\" (UID: \"839f94be-9343-4fc5-830b-d489bbdb4059\") " Oct 30 19:45:51.913778 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913732 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:51.913847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.913792 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:51.914116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.914090 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:51.914288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.914122 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:51.914417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.914401 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:51.914539 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.914514 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:51.914645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.914544 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:51.915770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.915746 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:51.915954 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.915844 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:51.916028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.915997 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:51.917261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:51.917239 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj" pod "839f94be-9343-4fc5-830b-d489bbdb4059" (UID: "839f94be-9343-4fc5-830b-d489bbdb4059"). InnerVolumeSpecName "kube-api-access-sv9zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:52.016171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.016144 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839f94be-9343-4fc5-830b-d489bbdb4059-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.016171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.016170 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.016354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.016181 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839f94be-9343-4fc5-830b-d489bbdb4059-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.016354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.016192 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9zj\" (UniqueName: \"kubernetes.io/projected/839f94be-9343-4fc5-830b-d489bbdb4059-kube-api-access-sv9zj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.016354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.016203 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839f94be-9343-4fc5-830b-d489bbdb4059-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.487020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.486981 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:52.686999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.686966 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" event={"ID":"839f94be-9343-4fc5-830b-d489bbdb4059","Type":"ContainerDied","Data":"5eb366c6fd70e58474019f7e5556fb7ebbf73db734505de86118bc935b89b425"} Oct 30 19:45:52.687447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.687022 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n" Oct 30 19:45:52.687447 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.687026 6514 scope.go:122] "RemoveContainer" containerID="1c3ba15c970e28e9b1b44493207109256d46d1a496784b86bf66f70401d5a2d0" Oct 30 19:45:52.726676 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.726655 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n"] Oct 30 19:45:52.737317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:52.737270 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4g9n"] Oct 30 19:45:53.190967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:53.190938 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp"] Oct 30 19:45:53.206382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:53.206357 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d"] Oct 30 19:45:53.403936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:53.403906 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:53.787366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:53.784954 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:54.157950 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:54.157830 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" path="/var/lib/kubelet/pods/839f94be-9343-4fc5-830b-d489bbdb4059/volumes" Oct 30 19:45:54.270621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:54.270568 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" containerName="payload-processing" containerID="cri-o://09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18" gracePeriod=25 Oct 30 19:45:54.297322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:54.297288 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" podUID="428a3c12-c612-40b0-a039-b08536c54541" containerName="payload-pre-processing" containerID="cri-o://0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127" gracePeriod=25 Oct 30 19:45:55.140159 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:55.140127 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:55.979234 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:55.979203 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.769901 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:56.769855 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:57.655224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:57.655187 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:58.465621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:58.465586 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:58.491887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:58.491481 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9"] Oct 30 19:45:58.646184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:58.646157 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:58.829052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:45:58.828966 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.599910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.598391 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt"] Oct 30 19:46:01.599910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.598724 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" podUID="9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" containerName="maas-api" containerID="cri-o://a28bd39fa4d3fe20eb2ebc81ca6412d9ae545ff8dd0e139038791d60c44c608a" gracePeriod=30 Oct 30 19:46:01.692240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.692205 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.742067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.742038 6514 generic.go:356] "Generic (PLEG): container finished" podID="9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" containerID="a28bd39fa4d3fe20eb2ebc81ca6412d9ae545ff8dd0e139038791d60c44c608a" exitCode=0 Oct 30 19:46:01.742243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.742105 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" event={"ID":"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a","Type":"ContainerDied","Data":"a28bd39fa4d3fe20eb2ebc81ca6412d9ae545ff8dd0e139038791d60c44c608a"} Oct 30 19:46:01.890783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.890756 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.946459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:01.946168 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:46:02.014372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.014325 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.038438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.038408 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls\") pod \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " Oct 30 19:46:02.038605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.038452 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca\") pod \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " Oct 30 19:46:02.038605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.038543 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs\") pod \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " Oct 30 19:46:02.038605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.038582 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9\" (UniqueName: \"kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9\") pod \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\" (UID: \"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a\") " Oct 30 19:46:02.038927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.038898 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca" pod "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" (UID: "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:02.039125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.039107 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:02.040655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.040624 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls" pod "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" (UID: "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:02.040746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.040663 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs" pod "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" (UID: "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:02.041039 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.041019 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9" pod "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" (UID: "9aa7fbc4-fbe9-4d60-90d5-00afd36a947a"). InnerVolumeSpecName "kube-api-access-mbwn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:02.140681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.140596 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:02.140681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.140629 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:02.140681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.140640 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwn9\" (UniqueName: \"kubernetes.io/projected/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a-kube-api-access-mbwn9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:02.487680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.487639 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.748399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.748324 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" Oct 30 19:46:02.748399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.748360 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt" event={"ID":"9aa7fbc4-fbe9-4d60-90d5-00afd36a947a","Type":"ContainerDied","Data":"a553cbc0905590179f1e2ab596d0594b8065e365dc9ace913dc6ca7c88fd4011"} Oct 30 19:46:02.748836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.748425 6514 scope.go:122] "RemoveContainer" containerID="a28bd39fa4d3fe20eb2ebc81ca6412d9ae545ff8dd0e139038791d60c44c608a" Oct 30 19:46:02.783484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.783458 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt"] Oct 30 19:46:02.788836 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.788813 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ce69f3-86546579fb-hpvmt"] Oct 30 19:46:02.898050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:02.898022 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:46:03.304740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:03.304704 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" podUID="428a3c12-c612-40b0-a039-b08536c54541" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:03.304935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:03.304707 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:03.492753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:03.492710 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerName="payload-processing" containerID="cri-o://bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f" gracePeriod=25 Oct 30 19:46:04.160838 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:04.160795 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" path="/var/lib/kubelet/pods/9aa7fbc4-fbe9-4d60-90d5-00afd36a947a/volumes" Oct 30 19:46:07.345946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:07.345908 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:08.211435 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.211397 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" containerID="cri-o://e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780" gracePeriod=45 Oct 30 19:46:08.466145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.466076 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:46:08.531471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531432 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir\") pod \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " Oct 30 19:46:08.531651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531511 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home\") pod \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " Oct 30 19:46:08.531651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531549 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs\") pod \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " Oct 30 19:46:08.531651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531606 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm\") pod \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " Oct 30 19:46:08.531931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531694 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir" pod "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" (UID: "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.531931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531790 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl\" (UniqueName: \"kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl\") pod \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " Oct 30 19:46:08.531931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531798 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home" pod "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" (UID: "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.531931 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.531824 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache\") pod \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\" (UID: \"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3\") " Oct 30 19:46:08.532313 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.532282 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache" pod "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" (UID: "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.532413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.532293 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.532413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.532344 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.533914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.533854 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm" pod "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" (UID: "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.534018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.533999 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl" pod "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" (UID: "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3"). InnerVolumeSpecName "kube-api-access-5vrnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:08.534195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.534179 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs" pod "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" (UID: "7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:08.649692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.649650 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrnl\" (UniqueName: \"kubernetes.io/projected/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-kube-api-access-5vrnl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.649692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.649694 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.650017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.649710 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.650017 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.649725 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.713412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.713366 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" containerID="cri-o://03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b" gracePeriod=45 Oct 30 19:46:08.789267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.789233 6514 generic.go:356] "Generic (PLEG): container finished" podID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerID="e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780" exitCode=0 Oct 30 19:46:08.789422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.789286 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" Oct 30 19:46:08.789422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.789322 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" event={"ID":"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3","Type":"ContainerDied","Data":"e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780"} Oct 30 19:46:08.789422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.789374 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp" event={"ID":"7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3","Type":"ContainerDied","Data":"9fc1aba02a2d915d9d06079b6383c0509f98a29abcd70f551ad7775ac544fac2"} Oct 30 19:46:08.789422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.789398 6514 scope.go:122] "RemoveContainer" containerID="e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780" Oct 30 19:46:08.821716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.821693 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp"] Oct 30 19:46:08.832221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.832162 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/premium-w4-ad96a4-kserve-56f77ddb75-p2pfp"] Oct 30 19:46:08.856885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.852552 6514 scope.go:122] "RemoveContainer" containerID="e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780" Oct 30 19:46:08.860881 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:08.860376 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780\": container with ID starting with e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780 not found: ID does not exist" containerID="e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780" Oct 30 19:46:08.860881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:08.860439 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780"} err="failed to get container status \"e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780\": rpc error: code = NotFound desc = could not find container \"e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780\": container with ID starting with e5dbc9b50ed894631603e1a608edbc01f9ef7b30425266cacbbc032587e69780 not found: ID does not exist" Oct 30 19:46:09.003264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.003241 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:46:09.053682 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053653 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v\" (UniqueName: \"kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v\") pod \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " Oct 30 19:46:09.053801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053685 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache\") pod \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " Oct 30 19:46:09.053801 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053762 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs\") pod \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " Oct 30 19:46:09.053915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053849 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm\") pod \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " Oct 30 19:46:09.053976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053915 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home\") pod \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " Oct 30 19:46:09.053976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053938 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir\") pod \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\" (UID: \"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d\") " Oct 30 19:46:09.054084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.053985 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache" pod "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" (UID: "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:09.054195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.054169 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home" pod "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" (UID: "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:09.054295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.054276 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.054353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.054301 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.054353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.054305 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir" pod "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" (UID: "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:09.055821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.055799 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v" pod "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" (UID: "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d"). InnerVolumeSpecName "kube-api-access-j549v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:09.055926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.055839 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs" pod "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" (UID: "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:09.055926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.055847 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm" pod "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" (UID: "d6be95ad-7c3d-4ab6-a4d0-47f523233a1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:09.155325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.155295 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.155325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.155324 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.155537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.155334 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j549v\" (UniqueName: \"kubernetes.io/projected/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-kube-api-access-j549v\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.155537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.155346 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.797385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.797345 6514 generic.go:356] "Generic (PLEG): container finished" podID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerID="03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b" exitCode=0 Oct 30 19:46:09.797802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.797399 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" Oct 30 19:46:09.797802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.797410 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" event={"ID":"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d","Type":"ContainerDied","Data":"03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b"} Oct 30 19:46:09.797802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.797449 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d" event={"ID":"d6be95ad-7c3d-4ab6-a4d0-47f523233a1d","Type":"ContainerDied","Data":"f3b0085996545f348834e3ba4a6799b4e8421c3095cf11a312cdab5cb458a543"} Oct 30 19:46:09.797802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.797472 6514 scope.go:122] "RemoveContainer" containerID="03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b" Oct 30 19:46:09.806708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.806684 6514 scope.go:122] "RemoveContainer" containerID="03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b" Oct 30 19:46:09.807024 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:09.807002 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b\": container with ID starting with 03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b not found: ID does not exist" containerID="03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b" Oct 30 19:46:09.807090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.807033 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b"} err="failed to get container status \"03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b\": rpc error: code = NotFound desc = could not find container \"03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b\": container with ID starting with 03ca419eab707bcdcec09067446c6a5993f75075c97f46dca4e10e916d3eb46b not found: ID does not exist" Oct 30 19:46:09.820206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.820181 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d"] Oct 30 19:46:09.822386 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:09.822359 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/simulator-w4-ad96a4-kserve-774fddb6fd-cxl2d"] Oct 30 19:46:10.157818 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.157746 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" path="/var/lib/kubelet/pods/7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3/volumes" Oct 30 19:46:10.158209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.158194 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" path="/var/lib/kubelet/pods/d6be95ad-7c3d-4ab6-a4d0-47f523233a1d/volumes" Oct 30 19:46:10.407940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.407840 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:10.539196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.539164 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:10.748715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.748682 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp"] Oct 30 19:46:10.748956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.748932 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" containerID="cri-o://981b3d088539f6d42a39a67bdc05d9a09f1dbc8b37deff5b4004eba0f71e094d" gracePeriod=30 Oct 30 19:46:10.755601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.755571 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm"] Oct 30 19:46:10.755795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.755772 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" containerID="cri-o://d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57" gracePeriod=30 Oct 30 19:46:10.824067 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:10.824037 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:11.609758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:11.609725 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:11.665614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:11.665579 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:46:11.720795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:11.720769 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.344074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.343197 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd"] Oct 30 19:46:12.344074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.343459 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" podUID="c6153b05-0bc3-410d-885d-30da599e0594" containerName="maas-api" containerID="cri-o://81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8" gracePeriod=30 Oct 30 19:46:12.427168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.427136 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.667563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.667386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.736833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.728032 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:46:12.778772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.778739 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.814969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.814232 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd\" (UniqueName: \"kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd\") pod \"c6153b05-0bc3-410d-885d-30da599e0594\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " Oct 30 19:46:12.830800 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.821339 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd" pod "c6153b05-0bc3-410d-885d-30da599e0594" (UID: "c6153b05-0bc3-410d-885d-30da599e0594"). InnerVolumeSpecName "kube-api-access-mxphd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:12.849678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.848840 6514 generic.go:356] "Generic (PLEG): container finished" podID="c6153b05-0bc3-410d-885d-30da599e0594" containerID="81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8" exitCode=0 Oct 30 19:46:12.849678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.848914 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" event={"ID":"c6153b05-0bc3-410d-885d-30da599e0594","Type":"ContainerDied","Data":"81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8"} Oct 30 19:46:12.849678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.848965 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" event={"ID":"c6153b05-0bc3-410d-885d-30da599e0594","Type":"ContainerDied","Data":"c2280b589d73977140201c35fa815b5231d10fba0e85bf1f1b1c8038dba11e15"} Oct 30 19:46:12.849678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.848994 6514 scope.go:122] "RemoveContainer" containerID="81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8" Oct 30 19:46:12.849678 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.849142 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd" Oct 30 19:46:12.866962 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.866938 6514 scope.go:122] "RemoveContainer" containerID="81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8" Oct 30 19:46:12.867318 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:12.867287 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8\": container with ID starting with 81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8 not found: ID does not exist" containerID="81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8" Oct 30 19:46:12.867394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.867332 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8"} err="failed to get container status \"81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8\": rpc error: code = NotFound desc = could not find container \"81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8\": container with ID starting with 81ce8cfcfae9ba2867b6b0c7330699bcaf09c4430465cc6beb23175b1ec6f0b8 not found: ID does not exist" Oct 30 19:46:12.915885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.915465 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs\") pod \"c6153b05-0bc3-410d-885d-30da599e0594\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " Oct 30 19:46:12.915885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.915522 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls\") pod \"c6153b05-0bc3-410d-885d-30da599e0594\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " Oct 30 19:46:12.915885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.915792 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca\") pod \"c6153b05-0bc3-410d-885d-30da599e0594\" (UID: \"c6153b05-0bc3-410d-885d-30da599e0594\") " Oct 30 19:46:12.919918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.916187 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxphd\" (UniqueName: \"kubernetes.io/projected/c6153b05-0bc3-410d-885d-30da599e0594-kube-api-access-mxphd\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:12.919918 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.916558 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca" pod "c6153b05-0bc3-410d-885d-30da599e0594" (UID: "c6153b05-0bc3-410d-885d-30da599e0594"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:12.923742 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.923684 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls" pod "c6153b05-0bc3-410d-885d-30da599e0594" (UID: "c6153b05-0bc3-410d-885d-30da599e0594"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:12.925204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:12.925168 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs" pod "c6153b05-0bc3-410d-885d-30da599e0594" (UID: "c6153b05-0bc3-410d-885d-30da599e0594"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:13.018114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.018033 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6153b05-0bc3-410d-885d-30da599e0594-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:13.018114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.018072 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:13.018114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.018090 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6153b05-0bc3-410d-885d-30da599e0594-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:13.178703 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.178678 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd"] Oct 30 19:46:13.183448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.183422 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad96a4-f99d9dfd-wcbwd"] Oct 30 19:46:13.311916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.311492 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:13.319020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.318990 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" podUID="428a3c12-c612-40b0-a039-b08536c54541" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:13.452499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:13.452421 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:14.157686 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.157653 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6153b05-0bc3-410d-885d-30da599e0594" path="/var/lib/kubelet/pods/c6153b05-0bc3-410d-885d-30da599e0594/volumes" Oct 30 19:46:14.206568 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.206536 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j"] Oct 30 19:46:14.206781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.206756 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" containerID="cri-o://c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571" gracePeriod=30 Oct 30 19:46:14.542629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.542604 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:46:14.575084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.575056 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:46:14.638122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.638096 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume\") pod \"428a3c12-c612-40b0-a039-b08536c54541\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " Oct 30 19:46:14.638249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.638158 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p\" (UniqueName: \"kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p\") pod \"f7802d20-9ba2-475f-8d33-05497773ba90\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " Oct 30 19:46:14.638249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.638211 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume\") pod \"f7802d20-9ba2-475f-8d33-05497773ba90\" (UID: \"f7802d20-9ba2-475f-8d33-05497773ba90\") " Oct 30 19:46:14.638340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.638297 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz\" (UniqueName: \"kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz\") pod \"428a3c12-c612-40b0-a039-b08536c54541\" (UID: \"428a3c12-c612-40b0-a039-b08536c54541\") " Oct 30 19:46:14.638611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.638583 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume" pod "428a3c12-c612-40b0-a039-b08536c54541" (UID: "428a3c12-c612-40b0-a039-b08536c54541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:14.638696 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.638637 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume" pod "f7802d20-9ba2-475f-8d33-05497773ba90" (UID: "f7802d20-9ba2-475f-8d33-05497773ba90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:14.640251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.640222 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p" pod "f7802d20-9ba2-475f-8d33-05497773ba90" (UID: "f7802d20-9ba2-475f-8d33-05497773ba90"). InnerVolumeSpecName "kube-api-access-kk62p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:14.640358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.640333 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz" pod "428a3c12-c612-40b0-a039-b08536c54541" (UID: "428a3c12-c612-40b0-a039-b08536c54541"). InnerVolumeSpecName "kube-api-access-58gdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:14.738984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.738959 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58gdz\" (UniqueName: \"kubernetes.io/projected/428a3c12-c612-40b0-a039-b08536c54541-kube-api-access-58gdz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:14.738984 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.738983 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a3c12-c612-40b0-a039-b08536c54541-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:14.739122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.738993 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk62p\" (UniqueName: \"kubernetes.io/projected/f7802d20-9ba2-475f-8d33-05497773ba90-kube-api-access-kk62p\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:14.739122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.739003 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7802d20-9ba2-475f-8d33-05497773ba90-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:14.862235 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.862207 6514 generic.go:356] "Generic (PLEG): container finished" podID="428a3c12-c612-40b0-a039-b08536c54541" containerID="0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127" exitCode=0 Oct 30 19:46:14.862365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.862253 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" Oct 30 19:46:14.862365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.862282 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" event={"ID":"428a3c12-c612-40b0-a039-b08536c54541","Type":"ContainerDied","Data":"0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127"} Oct 30 19:46:14.862365 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.862349 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc" event={"ID":"428a3c12-c612-40b0-a039-b08536c54541","Type":"ContainerDied","Data":"7fdfdc7f868deec844affed62d1adcbd54630ef2bb697fd9cf24f5dea0ce79e7"} Oct 30 19:46:14.862554 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.862384 6514 scope.go:122] "RemoveContainer" containerID="0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127" Oct 30 19:46:14.863699 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.863676 6514 generic.go:356] "Generic (PLEG): container finished" podID="f7802d20-9ba2-475f-8d33-05497773ba90" containerID="09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18" exitCode=0 Oct 30 19:46:14.863784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.863724 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" Oct 30 19:46:14.863784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.863757 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" event={"ID":"f7802d20-9ba2-475f-8d33-05497773ba90","Type":"ContainerDied","Data":"09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18"} Oct 30 19:46:14.863917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.863805 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw" event={"ID":"f7802d20-9ba2-475f-8d33-05497773ba90","Type":"ContainerDied","Data":"3ec09ac25b7ecd8c8d5161dcaf28f4b85939b967f361d8c15b8a2d574f3af889"} Oct 30 19:46:14.871366 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.871345 6514 scope.go:122] "RemoveContainer" containerID="0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127" Oct 30 19:46:14.871630 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:14.871607 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127\": container with ID starting with 0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127 not found: ID does not exist" containerID="0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127" Oct 30 19:46:14.871702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.871636 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127"} err="failed to get container status \"0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127\": rpc error: code = NotFound desc = could not find container \"0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127\": container with ID starting with 0d3fa0e445bf32d1f51cb6575192faa0cc8ecea5a9e1a475ec59476463ab5127 not found: ID does not exist" Oct 30 19:46:14.871702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.871654 6514 scope.go:122] "RemoveContainer" containerID="09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18" Oct 30 19:46:14.879755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.879738 6514 scope.go:122] "RemoveContainer" containerID="09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18" Oct 30 19:46:14.880044 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:14.880023 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18\": container with ID starting with 09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18 not found: ID does not exist" containerID="09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18" Oct 30 19:46:14.880117 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.880055 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18"} err="failed to get container status \"09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18\": rpc error: code = NotFound desc = could not find container \"09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18\": container with ID starting with 09f4cb1b8b29a1ff0b5478908a46ea5c594a22b3ef524e4c90fdbbbf33fdcf18 not found: ID does not exist" Oct 30 19:46:14.897046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.897027 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc"] Oct 30 19:46:14.902212 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.902190 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ce69f3-f897b54f5-hv6zc"] Oct 30 19:46:14.922229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.922206 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw"] Oct 30 19:46:14.928385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:14.928364 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ce69f3-598446f65b-z7qlw"] Oct 30 19:46:15.874203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:15.874099 6514 generic.go:356] "Generic (PLEG): container finished" podID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerID="981b3d088539f6d42a39a67bdc05d9a09f1dbc8b37deff5b4004eba0f71e094d" exitCode=0 Oct 30 19:46:15.874203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:15.874159 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" event={"ID":"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6","Type":"ContainerDied","Data":"981b3d088539f6d42a39a67bdc05d9a09f1dbc8b37deff5b4004eba0f71e094d"} Oct 30 19:46:16.020586 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.020529 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:46:16.035816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.035793 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:46:16.063385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063354 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.063519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063393 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.063519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063433 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.063519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063495 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.063707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063540 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.063707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063598 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.063707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063636 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.063707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063666 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.063707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063703 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063735 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063766 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063780 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063815 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063857 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063910 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063953 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063963 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.063958 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064017 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket\") pod \"c182ca0d-7923-47fd-9a77-1d468c8ece56\" (UID: \"c182ca0d-7923-47fd-9a77-1d468c8ece56\") " Oct 30 19:46:16.064057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064058 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.064582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064085 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy\") pod \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\" (UID: \"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6\") " Oct 30 19:46:16.064582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064189 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.064582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064536 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.064582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064572 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c182ca0d-7923-47fd-9a77-1d468c8ece56-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.064803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064587 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.064803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064601 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.064803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064605 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:16.065028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.064905 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.067820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.065104 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.067820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.066505 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.067820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.066742 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.067820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.067517 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.067820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.067531 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.067820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.067536 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:16.068699 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.068665 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.071139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.070195 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "kube-api-access-dgc8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.071139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.070233 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "kube-api-access-p8xdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.071139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.070390 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket" pod "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" (UID: "83f78ad1-02ff-4db7-90bf-46cf04ea1bd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.071559 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.071527 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:16.072395 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.072367 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token" pod "c182ca0d-7923-47fd-9a77-1d468c8ece56" (UID: "c182ca0d-7923-47fd-9a77-1d468c8ece56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.164517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.164480 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428a3c12-c612-40b0-a039-b08536c54541" path="/var/lib/kubelet/pods/428a3c12-c612-40b0-a039-b08536c54541/volumes" Oct 30 19:46:16.164893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.164879 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" path="/var/lib/kubelet/pods/f7802d20-9ba2-475f-8d33-05497773ba90/volumes" Oct 30 19:46:16.165388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165364 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165391 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165406 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165418 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165432 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8xdk\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-kube-api-access-p8xdk\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165446 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165460 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165472 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165485 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165497 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165511 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165525 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgc8l\" (UniqueName: \"kubernetes.io/projected/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-kube-api-access-dgc8l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165539 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c182ca0d-7923-47fd-9a77-1d468c8ece56-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.165842 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.165553 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.881078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.880713 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" Oct 30 19:46:16.881078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.880708 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp" event={"ID":"83f78ad1-02ff-4db7-90bf-46cf04ea1bd6","Type":"ContainerDied","Data":"42a50f666de15cb423a5ba76914d1eb7899db6c76860cf5ec9db93dc145e6ca9"} Oct 30 19:46:16.881078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.880943 6514 scope.go:122] "RemoveContainer" containerID="981b3d088539f6d42a39a67bdc05d9a09f1dbc8b37deff5b4004eba0f71e094d" Oct 30 19:46:16.883312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.883287 6514 generic.go:356] "Generic (PLEG): container finished" podID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerID="d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57" exitCode=0 Oct 30 19:46:16.883517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.883359 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" Oct 30 19:46:16.883517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.883367 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" event={"ID":"c182ca0d-7923-47fd-9a77-1d468c8ece56","Type":"ContainerDied","Data":"d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57"} Oct 30 19:46:16.883517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.883410 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm" event={"ID":"c182ca0d-7923-47fd-9a77-1d468c8ece56","Type":"ContainerDied","Data":"8f4019f4d8123632ddf2e2cf61cfe13110010d4d2151718c6d45a8eedcc47044"} Oct 30 19:46:16.891711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.891690 6514 scope.go:122] "RemoveContainer" containerID="d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57" Oct 30 19:46:16.899971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.899949 6514 scope.go:122] "RemoveContainer" containerID="d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57" Oct 30 19:46:16.900284 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:16.900256 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57\": container with ID starting with d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57 not found: ID does not exist" containerID="d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57" Oct 30 19:46:16.900382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.900294 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57"} err="failed to get container status \"d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57\": rpc error: code = NotFound desc = could not find container \"d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57\": container with ID starting with d87fa67ba62e451a508e5a44d2fd58909c49b6e12607a7168910c52aed5dff57 not found: ID does not exist" Oct 30 19:46:16.902484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.902459 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm"] Oct 30 19:46:16.908683 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.908662 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-whfvm"] Oct 30 19:46:16.921982 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.921960 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp"] Oct 30 19:46:16.927223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:16.927200 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-mn4mp"] Oct 30 19:46:17.346379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:17.346341 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:18.157881 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:18.157843 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" path="/var/lib/kubelet/pods/83f78ad1-02ff-4db7-90bf-46cf04ea1bd6/volumes" Oct 30 19:46:18.158305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:18.158291 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" path="/var/lib/kubelet/pods/c182ca0d-7923-47fd-9a77-1d468c8ece56/volumes" Oct 30 19:46:19.147018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.146984 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:46:19.457399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.457375 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:46:19.499977 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.499951 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500033 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500075 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500108 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500145 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500204 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500236 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500273 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500296 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:19.500511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500353 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data\") pod \"e106113d-7509-4af9-8a5f-adaf0039fc06\" (UID: \"e106113d-7509-4af9-8a5f-adaf0039fc06\") " Oct 30 19:46:19.500709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500683 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:19.500779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500704 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:19.500779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500704 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:19.500956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.500932 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:19.501132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.501108 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.501204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.501139 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.501204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.501155 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.501204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.501169 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e106113d-7509-4af9-8a5f-adaf0039fc06-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.501204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.501184 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.502223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.502202 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:19.502521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.502493 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "kube-api-access-64blt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:19.502599 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.502541 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:19.503150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.503134 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo" pod "e106113d-7509-4af9-8a5f-adaf0039fc06" (UID: "e106113d-7509-4af9-8a5f-adaf0039fc06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:19.602623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.602588 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.602623 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.602622 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64blt\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-kube-api-access-64blt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.602805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.602636 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.602805 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.602648 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e106113d-7509-4af9-8a5f-adaf0039fc06-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:19.904318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.904226 6514 generic.go:356] "Generic (PLEG): container finished" podID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerID="c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571" exitCode=0 Oct 30 19:46:19.904318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.904263 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" event={"ID":"e106113d-7509-4af9-8a5f-adaf0039fc06","Type":"ContainerDied","Data":"c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571"} Oct 30 19:46:19.904318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.904300 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" Oct 30 19:46:19.904602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.904320 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j" event={"ID":"e106113d-7509-4af9-8a5f-adaf0039fc06","Type":"ContainerDied","Data":"fd82fee7b3f300b9e62e553f1a95a30a74c4066c70470c3e9395bc52def6bfd6"} Oct 30 19:46:19.904602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.904348 6514 scope.go:122] "RemoveContainer" containerID="c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571" Oct 30 19:46:19.914498 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.914481 6514 scope.go:122] "RemoveContainer" containerID="c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571" Oct 30 19:46:19.914775 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:19.914748 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571\": container with ID starting with c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571 not found: ID does not exist" containerID="c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571" Oct 30 19:46:19.914827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.914788 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571"} err="failed to get container status \"c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571\": rpc error: code = NotFound desc = could not find container \"c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571\": container with ID starting with c44579df3c48626dde31cc6ae6de84d8bf55c0d95bf9b182e9ad33cdf260d571 not found: ID does not exist" Oct 30 19:46:19.932550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.932527 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j"] Oct 30 19:46:19.941976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:19.941956 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-jrr8j"] Oct 30 19:46:20.162727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:20.162656 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" path="/var/lib/kubelet/pods/e106113d-7509-4af9-8a5f-adaf0039fc06/volumes" Oct 30 19:46:20.893539 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:20.893501 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:46:21.039114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:21.039082 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:21.318732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:21.318626 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:21.407833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:21.407798 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb"] Oct 30 19:46:21.408091 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:21.408031 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" containerID="cri-o://f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5" gracePeriod=30 Oct 30 19:46:21.523230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:21.523191 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:21.748267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:21.748233 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:23.285987 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.285953 6514 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:46:23.286380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.286012 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:46:23.584771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.584703 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:23.744542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.744518 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:46:23.853690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.853600 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume\") pod \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " Oct 30 19:46:23.853848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.853737 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2\" (UniqueName: \"kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2\") pod \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\" (UID: \"0bcab47b-bfc9-4b92-8699-24cecb8a958a\") " Oct 30 19:46:23.853952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.853930 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume" pod "0bcab47b-bfc9-4b92-8699-24cecb8a958a" (UID: "0bcab47b-bfc9-4b92-8699-24cecb8a958a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:23.854075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.854061 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcab47b-bfc9-4b92-8699-24cecb8a958a-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:23.855878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.855841 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2" pod "0bcab47b-bfc9-4b92-8699-24cecb8a958a" (UID: "0bcab47b-bfc9-4b92-8699-24cecb8a958a"). InnerVolumeSpecName "kube-api-access-8qgz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:23.927597 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.927569 6514 generic.go:356] "Generic (PLEG): container finished" podID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerID="bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f" exitCode=0 Oct 30 19:46:23.927728 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.927615 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" Oct 30 19:46:23.927728 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.927657 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" event={"ID":"0bcab47b-bfc9-4b92-8699-24cecb8a958a","Type":"ContainerDied","Data":"bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f"} Oct 30 19:46:23.927728 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.927721 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9" event={"ID":"0bcab47b-bfc9-4b92-8699-24cecb8a958a","Type":"ContainerDied","Data":"d795ebe7d4ff71c43f78fca7451c8b33e4bdd553ddc1103cd332808acbd3d045"} Oct 30 19:46:23.927933 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.927755 6514 scope.go:122] "RemoveContainer" containerID="bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f" Oct 30 19:46:23.937316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.937297 6514 scope.go:122] "RemoveContainer" containerID="bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f" Oct 30 19:46:23.937583 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:23.937562 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f\": container with ID starting with bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f not found: ID does not exist" containerID="bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f" Oct 30 19:46:23.937667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.937589 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f"} err="failed to get container status \"bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f\": rpc error: code = NotFound desc = could not find container \"bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f\": container with ID starting with bc42f216a733bae779bf5c6633e2677806270e6822b8a395e1c9b0b82050f32f not found: ID does not exist" Oct 30 19:46:23.951124 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.951101 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9"] Oct 30 19:46:23.955182 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.955159 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgz2\" (UniqueName: \"kubernetes.io/projected/0bcab47b-bfc9-4b92-8699-24cecb8a958a-kube-api-access-8qgz2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:23.956249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:23.956231 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad96a4-7988fc87b-bpvc9"] Oct 30 19:46:24.156698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.156622 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" path="/var/lib/kubelet/pods/0bcab47b-bfc9-4b92-8699-24cecb8a958a/volumes" Oct 30 19:46:24.171615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.171589 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bb5597689-7xpv4"] Oct 30 19:46:24.171819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.171799 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6bb5597689-7xpv4" podUID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" containerName="manager" containerID="cri-o://f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6" gracePeriod=30 Oct 30 19:46:24.291128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.291090 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-6bb5597689-7xpv4" podUID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": dial tcp 10.134.0.26:8081: connect: connection refused" Oct 30 19:46:24.405709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.405684 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:46:24.459483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.459446 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8\" (UniqueName: \"kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8\") pod \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " Oct 30 19:46:24.459639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.459537 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs\") pod \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " Oct 30 19:46:24.459639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.459589 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert\") pod \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\" (UID: \"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5\") " Oct 30 19:46:24.462241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.462207 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs" pod "cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" (UID: "cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:24.462370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.462239 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8" pod "cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" (UID: "cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5"). InnerVolumeSpecName "kube-api-access-dfkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:24.462444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.462395 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert" pod "cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" (UID: "cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:24.561541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.561508 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkh8\" (UniqueName: \"kubernetes.io/projected/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-kube-api-access-dfkh8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:24.561541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.561539 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:24.561765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.561549 6514 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5-webhook-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:24.934149 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.934120 6514 generic.go:356] "Generic (PLEG): container finished" podID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" containerID="f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6" exitCode=0 Oct 30 19:46:24.934329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.934173 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-7xpv4" Oct 30 19:46:24.934329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.934201 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-7xpv4" event={"ID":"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5","Type":"ContainerDied","Data":"f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6"} Oct 30 19:46:24.934329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.934263 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-7xpv4" event={"ID":"cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5","Type":"ContainerDied","Data":"41dd8d1520dd52ea05466d78fa6a8d26b87ebd1879609d7c2d2098ee7f7397e5"} Oct 30 19:46:24.934329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.934294 6514 scope.go:122] "RemoveContainer" containerID="f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6" Oct 30 19:46:24.944553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.944537 6514 scope.go:122] "RemoveContainer" containerID="f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6" Oct 30 19:46:24.944813 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:24.944794 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6\": container with ID starting with f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6 not found: ID does not exist" containerID="f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6" Oct 30 19:46:24.944887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.944821 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6"} err="failed to get container status \"f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6\": rpc error: code = NotFound desc = could not find container \"f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6\": container with ID starting with f0ab0715eca76972f132fa98e0ad8fe0179708583d67aa4df2656c1bfc46e5a6 not found: ID does not exist" Oct 30 19:46:24.962221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.962197 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bb5597689-7xpv4"] Oct 30 19:46:24.967432 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:24.967410 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bb5597689-7xpv4"] Oct 30 19:46:25.734342 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734303 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bb5597689-f8vwt"] Oct 30 19:46:25.734908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734889 6514 cpu_manager.go:414] "containerMap: removing container" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" containerName="payload-processing" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734910 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" containerName="payload-processing" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734930 6514 cpu_manager.go:414] "containerMap: removing container" podUID="a8463f9f-e220-4d14-835c-2ae558cb209a" containerName="payload-pre-processing" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734938 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8463f9f-e220-4d14-835c-2ae558cb209a" containerName="payload-pre-processing" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734952 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" containerName="payload-pre-processing" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734959 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" containerName="payload-pre-processing" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734975 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734982 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734992 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" Oct 30 19:46:25.735001 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.734999 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735012 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735019 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735030 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735037 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735049 6514 cpu_manager.go:414] "containerMap: removing container" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735056 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735067 6514 cpu_manager.go:414] "containerMap: removing container" podUID="93e751e3-0511-411a-a275-cb9359879170" containerName="maas-api" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735080 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e751e3-0511-411a-a275-cb9359879170" containerName="maas-api" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735090 6514 cpu_manager.go:414] "containerMap: removing container" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735096 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735107 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735113 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735123 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e4bf65bc-6c93-48e4-941f-a7ac20801141" containerName="main" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735129 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bf65bc-6c93-48e4-941f-a7ac20801141" containerName="main" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735140 6514 cpu_manager.go:414] "containerMap: removing container" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735148 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735159 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735166 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735178 6514 cpu_manager.go:414] "containerMap: removing container" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735184 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735196 6514 cpu_manager.go:414] "containerMap: removing container" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735203 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735215 6514 cpu_manager.go:414] "containerMap: removing container" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735222 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735239 6514 cpu_manager.go:414] "containerMap: removing container" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735246 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735256 6514 cpu_manager.go:414] "containerMap: removing container" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735263 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735282 6514 cpu_manager.go:414] "containerMap: removing container" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735289 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735299 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735307 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735317 6514 cpu_manager.go:414] "containerMap: removing container" podUID="2552fab8-b507-4c24-b3a4-fd1e06c51835" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735324 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2552fab8-b507-4c24-b3a4-fd1e06c51835" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735336 6514 cpu_manager.go:414] "containerMap: removing container" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735343 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735353 6514 cpu_manager.go:414] "containerMap: removing container" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735361 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735372 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735379 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735391 6514 cpu_manager.go:414] "containerMap: removing container" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735398 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerName="cleanup" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735411 6514 cpu_manager.go:414] "containerMap: removing container" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735418 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735428 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4d9968f0-efba-4537-8f96-359415fa147f" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735435 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9968f0-efba-4537-8f96-359415fa147f" containerName="payload-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735444 6514 cpu_manager.go:414] "containerMap: removing container" podUID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" containerName="manager" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735451 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" containerName="manager" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735460 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c5c6654f-adc5-4091-a7b2-40af102bb695" containerName="maas-api" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735468 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c6654f-adc5-4091-a7b2-40af102bb695" containerName="maas-api" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735479 6514 cpu_manager.go:414] "containerMap: removing container" podUID="428a3c12-c612-40b0-a039-b08536c54541" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735486 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428a3c12-c612-40b0-a039-b08536c54541" containerName="payload-pre-processing" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735504 6514 cpu_manager.go:414] "containerMap: removing container" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" Oct 30 19:46:25.735487 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735511 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735523 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735530 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735543 6514 cpu_manager.go:414] "containerMap: removing container" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735549 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735564 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c6153b05-0bc3-410d-885d-30da599e0594" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735571 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6153b05-0bc3-410d-885d-30da599e0594" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735591 6514 cpu_manager.go:414] "containerMap: removing container" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735597 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735718 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f78ad1-02ff-4db7-90bf-46cf04ea1bd6" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735730 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e751e3-0511-411a-a275-cb9359879170" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735749 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735759 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c930635-2467-4bc3-94fb-80f8bf3361c8" containerName="payload-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735769 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bf65bc-6c93-48e4-941f-a7ac20801141" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735777 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735787 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d975d56-a1b1-4cdf-9dd1-3691ccabafbe" containerName="payload-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735801 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735809 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="984bd045-8081-46a7-a3a2-897b4210b861" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735817 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerName="cleanup" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735827 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfccb13-0b2b-4acd-85ad-6fb2255cb1f3" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735836 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="839f94be-9343-4fc5-830b-d489bbdb4059" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735848 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="269a3f6e-538b-4be8-8a51-b056342c28fe" containerName="payload-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735856 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6f8ff3-46bd-4617-9ed0-4a6aa4b226ab" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735896 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="2552fab8-b507-4c24-b3a4-fd1e06c51835" containerName="payload-pre-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735907 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15ed13e-ac83-4499-b154-05c89333c29c" containerName="payload-pre-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735916 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4b2279-7bef-4b97-ab14-28417776470e" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735925 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed81f2a-e68a-42ab-b845-758d39b5fbae" containerName="payload-pre-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735936 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf20b59-5da1-4cc2-9480-c178b6c94d7b" containerName="cleanup" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735946 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="45155a92-257c-47b0-a18e-38bff73f2120" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735954 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9968f0-efba-4537-8f96-359415fa147f" containerName="payload-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735963 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" containerName="manager" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735971 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c6654f-adc5-4091-a7b2-40af102bb695" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735978 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="428a3c12-c612-40b0-a039-b08536c54541" containerName="payload-pre-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.735991 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6be95ad-7c3d-4ab6-a4d0-47f523233a1d" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736000 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa7fbc4-fbe9-4d60-90d5-00afd36a947a" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736011 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d8af65-c587-4d75-8161-b40d13e320a1" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736021 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6153b05-0bc3-410d-885d-30da599e0594" containerName="maas-api" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736032 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b58f0f0-0f6a-4189-92c0-0921200c5c3d" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736043 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7802d20-9ba2-475f-8d33-05497773ba90" containerName="payload-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736052 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8463f9f-e220-4d14-835c-2ae558cb209a" containerName="payload-pre-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736061 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19b0962-4976-45a6-bffa-06720cab3b25" containerName="payload-pre-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736071 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acb067e-d374-4017-95c6-761dcde158cb" containerName="main" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736080 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c182ca0d-7923-47fd-9a77-1d468c8ece56" containerName="istio-proxy" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736089 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcab47b-bfc9-4b92-8699-24cecb8a958a" containerName="payload-processing" Oct 30 19:46:25.738028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.736099 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="e106113d-7509-4af9-8a5f-adaf0039fc06" containerName="istio-proxy" Oct 30 19:46:25.740741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.740720 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.765191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.765166 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5rr7g\"" Oct 30 19:46:25.765294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.765213 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:46:25.765336 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.765304 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:46:25.769207 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.769132 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:46:25.771141 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.771041 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bb5597689-f8vwt"] Oct 30 19:46:25.875274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.875238 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.875456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.875397 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zxt\" (UniqueName: \"kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.875505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.875462 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.977010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.976958 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.977192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.977046 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.977192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.977130 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zxt\" (UniqueName: \"kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.979709 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.979676 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.980316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.980295 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:25.989765 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:25.989701 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zxt\" (UniqueName: \"kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt\") pod \"maas-controller-6bb5597689-f8vwt\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:26.054269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.054237 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:26.157582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.157540 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5" path="/var/lib/kubelet/pods/cd81d55c-1bb0-4b71-8adb-0e1ccaf3efc5/volumes" Oct 30 19:46:26.337789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.337763 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bb5597689-f8vwt"] Oct 30 19:46:26.340737 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:46:26.340708 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b86377_d235_42dd_94d9_520995f5dde1.slice/crio-8701016d4a0b785649736ddaf18af22a56c205b62cfd4cade54b14e4ea1af5a7 WatchSource:0}: Error finding container 8701016d4a0b785649736ddaf18af22a56c205b62cfd4cade54b14e4ea1af5a7: Status 404 returned error can't find the container with id 8701016d4a0b785649736ddaf18af22a56c205b62cfd4cade54b14e4ea1af5a7 Oct 30 19:46:26.342763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.342740 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:46:26.691270 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.691240 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:46:26.785564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785529 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785579 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785603 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785645 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785676 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785698 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785813 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785842 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785916 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.785957 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket\") pod \"8b6b533e-4850-4148-b43e-a3504dbf8d85\" (UID: \"8b6b533e-4850-4148-b43e-a3504dbf8d85\") " Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786061 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.786088 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786076 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786147 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786346 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786484 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786501 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786512 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786521 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.786746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.786529 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b533e-4850-4148-b43e-a3504dbf8d85-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.788020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.787996 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.788130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.788069 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:26.788388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.788365 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:26.788388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.788378 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6" pod "8b6b533e-4850-4148-b43e-a3504dbf8d85" (UID: "8b6b533e-4850-4148-b43e-a3504dbf8d85"). InnerVolumeSpecName "kube-api-access-2btv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:26.888113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.888087 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.888113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.888112 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.888278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.888122 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btv6\" (UniqueName: \"kubernetes.io/projected/8b6b533e-4850-4148-b43e-a3504dbf8d85-kube-api-access-2btv6\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.888278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.888132 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b533e-4850-4148-b43e-a3504dbf8d85-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.957024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.952364 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-f8vwt" event={"ID":"48b86377-d235-42dd-94d9-520995f5dde1","Type":"ContainerStarted","Data":"8701016d4a0b785649736ddaf18af22a56c205b62cfd4cade54b14e4ea1af5a7"} Oct 30 19:46:26.957024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.954655 6514 generic.go:356] "Generic (PLEG): container finished" podID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerID="f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5" exitCode=0 Oct 30 19:46:26.957024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.954742 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" event={"ID":"8b6b533e-4850-4148-b43e-a3504dbf8d85","Type":"ContainerDied","Data":"f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5"} Oct 30 19:46:26.957024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.954784 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" event={"ID":"8b6b533e-4850-4148-b43e-a3504dbf8d85","Type":"ContainerDied","Data":"100fbd401ccd2f28b6772372417e836de735fc26eac1ad8813878885f08e6d47"} Oct 30 19:46:26.957024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.954815 6514 scope.go:122] "RemoveContainer" containerID="f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5" Oct 30 19:46:26.957024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.954981 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb" Oct 30 19:46:26.968789 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.968751 6514 scope.go:122] "RemoveContainer" containerID="f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5" Oct 30 19:46:26.969098 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:26.969077 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5\": container with ID starting with f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5 not found: ID does not exist" containerID="f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5" Oct 30 19:46:26.969168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.969107 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5"} err="failed to get container status \"f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5\": rpc error: code = NotFound desc = could not find container \"f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5\": container with ID starting with f65d18f332068a21ff49cf95d926b7a8df264651b93cab75401e1b172ad0e3c5 not found: ID does not exist" Oct 30 19:46:26.986296 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.986267 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb"] Oct 30 19:46:26.998897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:26.993333 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-8b2zb"] Oct 30 19:46:27.866045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:27.866007 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:46:27.866550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:27.866160 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:27.965010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:27.964969 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-f8vwt" event={"ID":"48b86377-d235-42dd-94d9-520995f5dde1","Type":"ContainerStarted","Data":"89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234"} Oct 30 19:46:27.965216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:27.965045 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:27.992526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:27.992458 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6bb5597689-f8vwt" podStartSLOduration=2.488184974 podStartE2EDuration="2.992437964s" totalImagesPullingTime="504.25299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:27.989574934 +0000 UTC m=+1757.947394123" watchObservedRunningTime="2026-10-30 19:46:27.992437964 +0000 UTC m=+1757.950257153" Oct 30 19:46:28.156831 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:28.156762 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" path="/var/lib/kubelet/pods/8b6b533e-4850-4148-b43e-a3504dbf8d85/volumes" Oct 30 19:46:29.704580 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:29.704532 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:29.877191 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:29.877151 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.004893 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:30.004790 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.204150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:30.204115 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.291615 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:31.291574 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.895231 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:31.895201 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.872255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:34.872215 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:35.066329 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:35.066292 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.898217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:37.898186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.973758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:38.973727 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:43.377925 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:43.377891 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:44.942062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942029 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5"] Oct 30 19:46:44.942456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942442 6514 cpu_manager.go:414] "containerMap: removing container" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" Oct 30 19:46:44.942503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942457 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" Oct 30 19:46:44.942503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942465 6514 cpu_manager.go:414] "containerMap: removing container" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:44.942503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942471 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:44.942503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942495 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" Oct 30 19:46:44.942503 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942501 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" Oct 30 19:46:44.942662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942567 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6b533e-4850-4148-b43e-a3504dbf8d85" containerName="istio-proxy" Oct 30 19:46:44.942662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942581 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="78eb4588-c43b-4925-99c5-73c0183e10ba" containerName="istio-proxy" Oct 30 19:46:44.942662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.942588 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="400fd1c0-4622-455d-abe4-752fded45a70" containerName="cleanup" Oct 30 19:46:44.945761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.945745 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:44.952023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.951990 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:46:44.953624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.953596 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:46:44.953758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.953645 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:46:44.953758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.953735 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:46:44.955575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.955550 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5"] Oct 30 19:46:44.956110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.956083 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnktf\" (UniqueName: \"kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:44.956247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.956229 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:44.956354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:44.956336 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.056798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.056768 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.057030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.056829 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnktf\" (UniqueName: \"kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.057030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.056938 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.057467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.057438 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.059280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.059254 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.066251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.066227 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnktf\" (UniqueName: \"kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf\") pod \"maas-api-revoke-sub-85237e45-xx5r5\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.260429 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.260341 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:45.541215 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:45.541189 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5"] Oct 30 19:46:45.543500 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:46:45.543469 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4556a91f_9de2_4309_b3b5_dfbb749dd8b2.slice/crio-49cb9ff9f4ccb3b6af75e66247b37323ad6d29b2a9136fc47cfec472bbdf7485 WatchSource:0}: Error finding container 49cb9ff9f4ccb3b6af75e66247b37323ad6d29b2a9136fc47cfec472bbdf7485: Status 404 returned error can't find the container with id 49cb9ff9f4ccb3b6af75e66247b37323ad6d29b2a9136fc47cfec472bbdf7485 Oct 30 19:46:46.061741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.061658 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" event={"ID":"4556a91f-9de2-4309-b3b5-dfbb749dd8b2","Type":"ContainerStarted","Data":"c7b6a87b6072bc74f71a2e75e21d47a13cd189a15072b1b013cec1b6d11731e4"} Oct 30 19:46:46.061741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.061717 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" event={"ID":"4556a91f-9de2-4309-b3b5-dfbb749dd8b2","Type":"ContainerStarted","Data":"49cb9ff9f4ccb3b6af75e66247b37323ad6d29b2a9136fc47cfec472bbdf7485"} Oct 30 19:46:46.081283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.081231 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" podStartSLOduration=2.081215992 podStartE2EDuration="2.081215992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:46.079781872 +0000 UTC m=+1776.037601051" watchObservedRunningTime="2026-10-30 19:46:46.081215992 +0000 UTC m=+1776.039035172" Oct 30 19:46:46.612917 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.612879 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn"] Oct 30 19:46:46.616279 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.616260 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.629910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.629882 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn"] Oct 30 19:46:46.655465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.655432 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv"] Oct 30 19:46:46.659412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.659385 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.669798 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.669761 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv"] Oct 30 19:46:46.671420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671393 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdxs\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.671565 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671497 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.671663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671565 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.671663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671598 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.671772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671663 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.671772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671695 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.671772 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671750 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.671919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671817 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.671919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671894 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.672031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.671979 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.672031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672019 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.672128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672044 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6ch\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.672170 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672137 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.672216 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672195 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.672320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672298 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.672396 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672379 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.672456 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672425 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.672521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.672468 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.773534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773503 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.773725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773540 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdxs\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.773725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773564 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.773725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773617 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.773725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773648 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.773725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773669 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.773725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773685 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773737 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773767 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773810 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.773829 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774123 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774064 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774154 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774169 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774186 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6ch\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774203 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774232 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774256 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774266 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774294 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774330 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774416 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774455 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.774919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774721 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.774919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774743 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.775041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.774991 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.776792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.775926 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.776792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.776710 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.777022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.776997 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.777073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.777051 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.781781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.777800 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.785335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.784090 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.785335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.784305 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.786579 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.786558 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdxs\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-742kv\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:46.790884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.790840 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6ch\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.790979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.790928 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s6ldn\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.930782 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.930757 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:46.975253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:46.975218 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:47.069921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.069886 6514 generic.go:356] "Generic (PLEG): container finished" podID="4556a91f-9de2-4309-b3b5-dfbb749dd8b2" containerID="c7b6a87b6072bc74f71a2e75e21d47a13cd189a15072b1b013cec1b6d11731e4" exitCode=0 Oct 30 19:46:47.070304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.070006 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" event={"ID":"4556a91f-9de2-4309-b3b5-dfbb749dd8b2","Type":"ContainerDied","Data":"c7b6a87b6072bc74f71a2e75e21d47a13cd189a15072b1b013cec1b6d11731e4"} Oct 30 19:46:47.228183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.228144 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn"] Oct 30 19:46:47.229250 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:46:47.229222 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e13b5d_c7da_408f_9557_866bc278760e.slice/crio-1c15aa84169ff8d0304d544975d773dea62fd53825ce22adf8a439fc8ff85c26 WatchSource:0}: Error finding container 1c15aa84169ff8d0304d544975d773dea62fd53825ce22adf8a439fc8ff85c26: Status 404 returned error can't find the container with id 1c15aa84169ff8d0304d544975d773dea62fd53825ce22adf8a439fc8ff85c26 Oct 30 19:46:47.232010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.231976 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.232102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.232044 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.232102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.232072 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.272448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.272263 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv"] Oct 30 19:46:47.279421 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:46:47.279355 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf818a7e_9582_43c4_ac90_2cf03bb9ddbd.slice/crio-abd9c582a9a7f1df462b6f289193feca9296af7c0d75545c628e8dd1943e0a47 WatchSource:0}: Error finding container abd9c582a9a7f1df462b6f289193feca9296af7c0d75545c628e8dd1943e0a47: Status 404 returned error can't find the container with id abd9c582a9a7f1df462b6f289193feca9296af7c0d75545c628e8dd1943e0a47 Oct 30 19:46:47.281923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.281890 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.282008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.281956 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:47.282008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:47.281988 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:48.076929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.076815 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" event={"ID":"99e13b5d-c7da-408f-9557-866bc278760e","Type":"ContainerStarted","Data":"0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529"} Oct 30 19:46:48.076929 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.076895 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" event={"ID":"99e13b5d-c7da-408f-9557-866bc278760e","Type":"ContainerStarted","Data":"1c15aa84169ff8d0304d544975d773dea62fd53825ce22adf8a439fc8ff85c26"} Oct 30 19:46:48.078510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.078474 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" event={"ID":"df818a7e-9582-43c4-ac90-2cf03bb9ddbd","Type":"ContainerStarted","Data":"0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a"} Oct 30 19:46:48.078629 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.078527 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" event={"ID":"df818a7e-9582-43c4-ac90-2cf03bb9ddbd","Type":"ContainerStarted","Data":"abd9c582a9a7f1df462b6f289193feca9296af7c0d75545c628e8dd1943e0a47"} Oct 30 19:46:48.120889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.110549 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podStartSLOduration=2.110529666 podStartE2EDuration="2.110529666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.100532376 +0000 UTC m=+1778.058351566" watchObservedRunningTime="2026-10-30 19:46:48.110529666 +0000 UTC m=+1778.068348856" Oct 30 19:46:48.351384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.351313 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:48.372090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.372029 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podStartSLOduration=2.372012154 podStartE2EDuration="2.372012154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.144218754 +0000 UTC m=+1778.102037944" watchObservedRunningTime="2026-10-30 19:46:48.372012154 +0000 UTC m=+1778.329831346" Oct 30 19:46:48.425887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.423225 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca\") pod \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " Oct 30 19:46:48.425887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.423359 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf\" (UniqueName: \"kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf\") pod \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " Oct 30 19:46:48.425887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.423479 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca\") pod \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\" (UID: \"4556a91f-9de2-4309-b3b5-dfbb749dd8b2\") " Oct 30 19:46:48.429891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.428353 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca" pod "4556a91f-9de2-4309-b3b5-dfbb749dd8b2" (UID: "4556a91f-9de2-4309-b3b5-dfbb749dd8b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:48.445915 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.445202 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca" pod "4556a91f-9de2-4309-b3b5-dfbb749dd8b2" (UID: "4556a91f-9de2-4309-b3b5-dfbb749dd8b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:48.447243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.447201 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf" pod "4556a91f-9de2-4309-b3b5-dfbb749dd8b2" (UID: "4556a91f-9de2-4309-b3b5-dfbb749dd8b2"). InnerVolumeSpecName "kube-api-access-dnktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:48.528889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.528431 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:48.528889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.528478 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:48.528889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.528494 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnktf\" (UniqueName: \"kubernetes.io/projected/4556a91f-9de2-4309-b3b5-dfbb749dd8b2-kube-api-access-dnktf\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:48.932889 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.932516 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:48.948884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.948743 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:48.948884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.948802 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:48.975964 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.975930 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:48.976771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.976748 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:48.976840 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:48.976793 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:49.084584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.084549 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" Oct 30 19:46:49.084997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.084584 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5" event={"ID":"4556a91f-9de2-4309-b3b5-dfbb749dd8b2","Type":"ContainerDied","Data":"49cb9ff9f4ccb3b6af75e66247b37323ad6d29b2a9136fc47cfec472bbdf7485"} Oct 30 19:46:49.084997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.084629 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49cb9ff9f4ccb3b6af75e66247b37323ad6d29b2a9136fc47cfec472bbdf7485" Oct 30 19:46:49.933888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.933334 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:49.933888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.933393 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:49.976442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.976405 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:49.976583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:49.976454 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:50.883461 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:50.883418 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:46:50.952886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:50.946473 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:50.952886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:50.946541 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:50.975740 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:50.975700 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:50.975960 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:50.975767 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:51.933886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:51.933171 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:51.933886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:51.933241 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:51.976161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:51.976124 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:51.976335 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:51.976187 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:52.931236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:52.931196 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:52.933675 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:52.933639 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:52.975854 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:52.975817 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:52.976330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:52.975891 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:53.931475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:53.931443 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:53.931645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:53.931496 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:53.975998 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:53.975956 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:53.976373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:53.976032 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:54.931450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:54.931411 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 19:46:54.931647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:54.931468 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 19:46:54.975911 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:54.975847 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:46:54.976116 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:54.975932 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 19:46:55.935692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:55.935661 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:55.936043 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:55.936023 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:55.936775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:55.936755 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:46:55.980302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:55.980263 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:55.980717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:55.980542 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:55.981284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:55.981266 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:46:56.530380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.530350 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:56.623406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.623370 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bb5597689-f8vwt"] Oct 30 19:46:56.623627 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.623604 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6bb5597689-f8vwt" podUID="48b86377-d235-42dd-94d9-520995f5dde1" containerName="manager" containerID="cri-o://89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234" gracePeriod=30 Oct 30 19:46:56.877996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.877967 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:56.938802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.938752 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs\") pod \"48b86377-d235-42dd-94d9-520995f5dde1\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " Oct 30 19:46:56.939042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.938854 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt\" (UniqueName: \"kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt\") pod \"48b86377-d235-42dd-94d9-520995f5dde1\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " Oct 30 19:46:56.939042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.938977 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert\") pod \"48b86377-d235-42dd-94d9-520995f5dde1\" (UID: \"48b86377-d235-42dd-94d9-520995f5dde1\") " Oct 30 19:46:56.942352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.942301 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt" pod "48b86377-d235-42dd-94d9-520995f5dde1" (UID: "48b86377-d235-42dd-94d9-520995f5dde1"). InnerVolumeSpecName "kube-api-access-n8zxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:56.942661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.942628 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs" pod "48b86377-d235-42dd-94d9-520995f5dde1" (UID: "48b86377-d235-42dd-94d9-520995f5dde1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:56.942810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:56.942687 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert" pod "48b86377-d235-42dd-94d9-520995f5dde1" (UID: "48b86377-d235-42dd-94d9-520995f5dde1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:57.041119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.041074 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8zxt\" (UniqueName: \"kubernetes.io/projected/48b86377-d235-42dd-94d9-520995f5dde1-kube-api-access-n8zxt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:57.041119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.041115 6514 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-webhook-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:57.041119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.041126 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48b86377-d235-42dd-94d9-520995f5dde1-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:57.171491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.171405 6514 generic.go:356] "Generic (PLEG): container finished" podID="48b86377-d235-42dd-94d9-520995f5dde1" containerID="89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234" exitCode=0 Oct 30 19:46:57.171491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.171463 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-f8vwt" Oct 30 19:46:57.171697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.171488 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-f8vwt" event={"ID":"48b86377-d235-42dd-94d9-520995f5dde1","Type":"ContainerDied","Data":"89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234"} Oct 30 19:46:57.171697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.171544 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-f8vwt" event={"ID":"48b86377-d235-42dd-94d9-520995f5dde1","Type":"ContainerDied","Data":"8701016d4a0b785649736ddaf18af22a56c205b62cfd4cade54b14e4ea1af5a7"} Oct 30 19:46:57.171697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.171567 6514 scope.go:122] "RemoveContainer" containerID="89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234" Oct 30 19:46:57.181111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.181087 6514 scope.go:122] "RemoveContainer" containerID="89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234" Oct 30 19:46:57.181383 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:46:57.181362 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234\": container with ID starting with 89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234 not found: ID does not exist" containerID="89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234" Oct 30 19:46:57.181458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.181391 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234"} err="failed to get container status \"89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234\": rpc error: code = NotFound desc = could not find container \"89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234\": container with ID starting with 89f8831c9b3c152c357aec3073a0b465a96baa575644b84802747e448b001234 not found: ID does not exist" Oct 30 19:46:57.195248 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.195218 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bb5597689-f8vwt"] Oct 30 19:46:57.202073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:57.202046 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bb5597689-f8vwt"] Oct 30 19:46:58.157905 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.157849 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b86377-d235-42dd-94d9-520995f5dde1" path="/var/lib/kubelet/pods/48b86377-d235-42dd-94d9-520995f5dde1/volumes" Oct 30 19:46:58.269243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.269212 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:58.407819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.407784 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bb5597689-4djsx"] Oct 30 19:46:58.408230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.408191 6514 cpu_manager.go:414] "containerMap: removing container" podUID="48b86377-d235-42dd-94d9-520995f5dde1" containerName="manager" Oct 30 19:46:58.408230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.408207 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b86377-d235-42dd-94d9-520995f5dde1" containerName="manager" Oct 30 19:46:58.408230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.408225 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4556a91f-9de2-4309-b3b5-dfbb749dd8b2" containerName="revoke-keys" Oct 30 19:46:58.408230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.408230 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4556a91f-9de2-4309-b3b5-dfbb749dd8b2" containerName="revoke-keys" Oct 30 19:46:58.408382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.408308 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b86377-d235-42dd-94d9-520995f5dde1" containerName="manager" Oct 30 19:46:58.408382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.408318 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4556a91f-9de2-4309-b3b5-dfbb749dd8b2" containerName="revoke-keys" Oct 30 19:46:58.412887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.412851 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.422146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.422118 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:46:58.423484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.423457 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5rr7g\"" Oct 30 19:46:58.423639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.423496 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:46:58.423639 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.423576 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:46:58.424162 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.424135 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bb5597689-4djsx"] Oct 30 19:46:58.556855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.556816 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfr8n\" (UniqueName: \"kubernetes.io/projected/bc34d546-8598-4841-a3e6-efb6011ac090-kube-api-access-nfr8n\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.556855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.556874 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc34d546-8598-4841-a3e6-efb6011ac090-webhook-cert\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.557090 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.556983 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc34d546-8598-4841-a3e6-efb6011ac090-metrics-certs\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.658214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.658176 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfr8n\" (UniqueName: \"kubernetes.io/projected/bc34d546-8598-4841-a3e6-efb6011ac090-kube-api-access-nfr8n\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.658214 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.658216 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc34d546-8598-4841-a3e6-efb6011ac090-webhook-cert\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.658488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.658240 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc34d546-8598-4841-a3e6-efb6011ac090-metrics-certs\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.660800 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.660771 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc34d546-8598-4841-a3e6-efb6011ac090-webhook-cert\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.660943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.660850 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc34d546-8598-4841-a3e6-efb6011ac090-metrics-certs\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.667571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.667551 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfr8n\" (UniqueName: \"kubernetes.io/projected/bc34d546-8598-4841-a3e6-efb6011ac090-kube-api-access-nfr8n\") pod \"maas-controller-6bb5597689-4djsx\" (UID: \"bc34d546-8598-4841-a3e6-efb6011ac090\") " pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:58.726271 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:58.726235 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:46:59.000097 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:59.000028 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bb5597689-4djsx"] Oct 30 19:46:59.002845 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:46:59.002816 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc34d546_8598_4841_a3e6_efb6011ac090.slice/crio-4b12b2f5e5018d8c5f8bd89244feaaf1831337e6bdbe81a4c023006f8b16aa98 WatchSource:0}: Error finding container 4b12b2f5e5018d8c5f8bd89244feaaf1831337e6bdbe81a4c023006f8b16aa98: Status 404 returned error can't find the container with id 4b12b2f5e5018d8c5f8bd89244feaaf1831337e6bdbe81a4c023006f8b16aa98 Oct 30 19:46:59.182727 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:46:59.182687 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-4djsx" event={"ID":"bc34d546-8598-4841-a3e6-efb6011ac090","Type":"ContainerStarted","Data":"4b12b2f5e5018d8c5f8bd89244feaaf1831337e6bdbe81a4c023006f8b16aa98"} Oct 30 19:47:00.188471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:00.188432 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bb5597689-4djsx" event={"ID":"bc34d546-8598-4841-a3e6-efb6011ac090","Type":"ContainerStarted","Data":"2c5add806d4ab5d1bb2cfde26108dcbf8c89e6d484042b7de849ccea3432d91a"} Oct 30 19:47:00.188938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:00.188525 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:47:00.207180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:00.207126 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6bb5597689-4djsx" podStartSLOduration=1.835365586 podStartE2EDuration="2.207110159s" totalImagesPullingTime="371.744573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:00.206095246 +0000 UTC m=+1790.163914436" watchObservedRunningTime="2026-10-30 19:47:00.207110159 +0000 UTC m=+1790.164929380" Oct 30 19:47:10.697374 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:10.697340 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:47:10.701301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:10.701279 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:47:11.195441 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:11.195411 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bb5597689-4djsx" Oct 30 19:47:12.677082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:12.677047 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:20.882045 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:47:20.882002 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:47:34.288970 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:34.288881 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:47:39.083054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.083021 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8"] Oct 30 19:47:39.086630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.086611 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.091356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.091332 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:47:39.091356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.091332 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:47:39.093021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.092989 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:39.093115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.093040 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:47:39.097062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.097041 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8"] Oct 30 19:47:39.140724 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.140685 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.140904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.140747 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8d8h\" (UniqueName: \"kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.140904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.140806 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.241879 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.241838 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.242062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.241919 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8d8h\" (UniqueName: \"kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.242062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.241958 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.242550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.242522 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.244225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.244205 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.251822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.251794 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8d8h\" (UniqueName: \"kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h\") pod \"maas-api-revoke-sub-c1e271a9-ngms8\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.402360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.402285 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:39.673940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:39.673915 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8"] Oct 30 19:47:39.675898 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:47:39.675847 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abbb50f_b8d8_40d1_b1bf_043ffd2c255c.slice/crio-25e96ba8e6cf81ed561e752e9093ed8bc73518363bad92cde3ca3e1d5036ec01 WatchSource:0}: Error finding container 25e96ba8e6cf81ed561e752e9093ed8bc73518363bad92cde3ca3e1d5036ec01: Status 404 returned error can't find the container with id 25e96ba8e6cf81ed561e752e9093ed8bc73518363bad92cde3ca3e1d5036ec01 Oct 30 19:47:40.403651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:40.403573 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" event={"ID":"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c","Type":"ContainerStarted","Data":"0c6d0563958ce7131e596d503997217fa2a6866b81ceacb4a0e7ab78b03895dd"} Oct 30 19:47:40.403651 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:40.403623 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" event={"ID":"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c","Type":"ContainerStarted","Data":"25e96ba8e6cf81ed561e752e9093ed8bc73518363bad92cde3ca3e1d5036ec01"} Oct 30 19:47:40.422345 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:40.422301 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" podStartSLOduration=1.4222878589999999 podStartE2EDuration="1.422287859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:40.421447807 +0000 UTC m=+1830.379266999" watchObservedRunningTime="2026-10-30 19:47:40.422287859 +0000 UTC m=+1830.380107040" Oct 30 19:47:41.409534 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:41.409498 6514 generic.go:356] "Generic (PLEG): container finished" podID="4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" containerID="0c6d0563958ce7131e596d503997217fa2a6866b81ceacb4a0e7ab78b03895dd" exitCode=0 Oct 30 19:47:41.410026 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:41.409587 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" event={"ID":"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c","Type":"ContainerDied","Data":"0c6d0563958ce7131e596d503997217fa2a6866b81ceacb4a0e7ab78b03895dd"} Oct 30 19:47:42.551232 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.551204 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:42.581553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.581522 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h\" (UniqueName: \"kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h\") pod \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " Oct 30 19:47:42.581732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.581613 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca\") pod \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " Oct 30 19:47:42.581732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.581644 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca\") pod \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\" (UID: \"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c\") " Oct 30 19:47:42.582012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.581988 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca" pod "4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" (UID: "4abbb50f-b8d8-40d1-b1bf-043ffd2c255c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:42.583614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.583590 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h" pod "4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" (UID: "4abbb50f-b8d8-40d1-b1bf-043ffd2c255c"). InnerVolumeSpecName "kube-api-access-x8d8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:42.583711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.583651 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca" pod "4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" (UID: "4abbb50f-b8d8-40d1-b1bf-043ffd2c255c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:42.683302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.683268 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:42.683302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.683298 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8d8h\" (UniqueName: \"kubernetes.io/projected/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-kube-api-access-x8d8h\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:42.683302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:42.683309 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:43.421506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:43.421472 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" event={"ID":"4abbb50f-b8d8-40d1-b1bf-043ffd2c255c","Type":"ContainerDied","Data":"25e96ba8e6cf81ed561e752e9093ed8bc73518363bad92cde3ca3e1d5036ec01"} Oct 30 19:47:43.421697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:43.421526 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e96ba8e6cf81ed561e752e9093ed8bc73518363bad92cde3ca3e1d5036ec01" Oct 30 19:47:43.421697 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:43.421482 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8" Oct 30 19:47:49.943419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:49.943385 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:47:50.888648 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:47:50.888611 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:47:51.150183 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:51.150093 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.266927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:51.266892 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.107725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:54.107682 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:47:57.832198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.832164 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t"] Oct 30 19:47:57.832647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.832629 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" containerName="revoke-keys" Oct 30 19:47:57.832647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.832647 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" containerName="revoke-keys" Oct 30 19:47:57.832820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.832727 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" containerName="revoke-keys" Oct 30 19:47:57.836809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.836751 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:57.844299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.844265 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:47:57.844663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.844637 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:57.845424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.844731 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:47:57.845424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.845226 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t"] Oct 30 19:47:57.847392 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.846780 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:47:57.974135 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.974104 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6thfg\" (UniqueName: \"kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:57.974303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.974155 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:57.974303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:57.974232 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.074821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.074775 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6thfg\" (UniqueName: \"kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.075046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.074835 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.075046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.074889 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.075553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.075521 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.077247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.077221 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.083809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.083758 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6thfg\" (UniqueName: \"kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg\") pod \"maas-api-revoke-sub-2e1f5b5f-x5w5t\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.154102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.154070 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:47:58.430853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.430668 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t"] Oct 30 19:47:58.433195 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:47:58.433170 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a6ee8c_7b77_4b54_96e2_ce8b84068a87.slice/crio-a25b9af4697b2771c16e867fc6b323c7f6441c7afacc2aad08d62d1914ba00f5 WatchSource:0}: Error finding container a25b9af4697b2771c16e867fc6b323c7f6441c7afacc2aad08d62d1914ba00f5: Status 404 returned error can't find the container with id a25b9af4697b2771c16e867fc6b323c7f6441c7afacc2aad08d62d1914ba00f5 Oct 30 19:47:58.509957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:58.509924 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" event={"ID":"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87","Type":"ContainerStarted","Data":"a25b9af4697b2771c16e867fc6b323c7f6441c7afacc2aad08d62d1914ba00f5"} Oct 30 19:47:59.515521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:59.515448 6514 generic.go:356] "Generic (PLEG): container finished" podID="b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" containerID="9051eeb1c4b34578c078690de58c1683e01a2d29dfa172aea4560e6d20110c99" exitCode=0 Oct 30 19:47:59.515957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:47:59.515537 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" event={"ID":"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87","Type":"ContainerDied","Data":"9051eeb1c4b34578c078690de58c1683e01a2d29dfa172aea4560e6d20110c99"} Oct 30 19:48:00.652525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.652490 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:48:00.803466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.803379 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca\") pod \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " Oct 30 19:48:00.803466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.803433 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg\" (UniqueName: \"kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg\") pod \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " Oct 30 19:48:00.803662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.803511 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca\") pod \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\" (UID: \"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87\") " Oct 30 19:48:00.803961 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.803934 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca" pod "b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" (UID: "b3a6ee8c-7b77-4b54-96e2-ce8b84068a87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:00.805618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.805593 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg" pod "b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" (UID: "b3a6ee8c-7b77-4b54-96e2-ce8b84068a87"). InnerVolumeSpecName "kube-api-access-6thfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:00.805618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.805599 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca" pod "b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" (UID: "b3a6ee8c-7b77-4b54-96e2-ce8b84068a87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:00.904853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.904814 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:00.904853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.904850 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:00.905084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:00.904889 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6thfg\" (UniqueName: \"kubernetes.io/projected/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87-kube-api-access-6thfg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:01.527065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:01.527034 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" Oct 30 19:48:01.527244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:01.527059 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t" event={"ID":"b3a6ee8c-7b77-4b54-96e2-ce8b84068a87","Type":"ContainerDied","Data":"a25b9af4697b2771c16e867fc6b323c7f6441c7afacc2aad08d62d1914ba00f5"} Oct 30 19:48:01.527244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:01.527112 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a25b9af4697b2771c16e867fc6b323c7f6441c7afacc2aad08d62d1914ba00f5" Oct 30 19:48:08.734151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.734113 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:48:08.857065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.857032 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46"] Oct 30 19:48:08.857442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.857428 6514 cpu_manager.go:414] "containerMap: removing container" podUID="b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" containerName="revoke-keys" Oct 30 19:48:08.857505 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.857442 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" containerName="revoke-keys" Oct 30 19:48:08.857567 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.857523 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" containerName="revoke-keys" Oct 30 19:48:08.860544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.860525 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:08.873380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.873353 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:08.873512 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.873429 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:08.874959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.874937 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:48:08.874959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.874950 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:08.881137 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.881094 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46"] Oct 30 19:48:08.982359 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.982326 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:08.982509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.982379 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwz9\" (UniqueName: \"kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:08.982509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:08.982500 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.083829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.083736 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.084168 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.084139 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwz9\" (UniqueName: \"kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.084583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.084551 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.084914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.084377 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.089317 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.089292 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.098165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.098127 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwz9\" (UniqueName: \"kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9\") pod \"maas-api-revoke-sub-ec843018-fgj46\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.173692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.173655 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:09.449957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.449930 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46"] Oct 30 19:48:09.452061 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:48:09.452037 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc8c4c1_61f1_443d_95c2_d1c694fe6bc3.slice/crio-85b9f86242e3c88561a11548c732bea3d7ffc1a93b4464599799f7dc544e2ff9 WatchSource:0}: Error finding container 85b9f86242e3c88561a11548c732bea3d7ffc1a93b4464599799f7dc544e2ff9: Status 404 returned error can't find the container with id 85b9f86242e3c88561a11548c732bea3d7ffc1a93b4464599799f7dc544e2ff9 Oct 30 19:48:09.566186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:09.566153 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" event={"ID":"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3","Type":"ContainerStarted","Data":"85b9f86242e3c88561a11548c732bea3d7ffc1a93b4464599799f7dc544e2ff9"} Oct 30 19:48:10.572458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:10.572383 6514 generic.go:356] "Generic (PLEG): container finished" podID="cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" containerID="f0e99493ddc43e8d9adb4d07e0910f16df659805b14727f12e4a420bba19d0ad" exitCode=0 Oct 30 19:48:10.572824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:10.572468 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" event={"ID":"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3","Type":"ContainerDied","Data":"f0e99493ddc43e8d9adb4d07e0910f16df659805b14727f12e4a420bba19d0ad"} Oct 30 19:48:11.787887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.785039 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:11.951886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.938201 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9\" (UniqueName: \"kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9\") pod \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " Oct 30 19:48:11.951886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.938276 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca\") pod \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " Oct 30 19:48:11.951886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.938363 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca\") pod \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\" (UID: \"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3\") " Oct 30 19:48:11.974899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.963889 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca" pod "cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" (UID: "cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:11.975069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.974978 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9" pod "cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" (UID: "cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3"). InnerVolumeSpecName "kube-api-access-9bwz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:11.979178 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:11.978965 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca" pod "cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" (UID: "cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:12.042887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.042066 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:12.042887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.042104 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwz9\" (UniqueName: \"kubernetes.io/projected/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-kube-api-access-9bwz9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:12.042887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.042121 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:12.589894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.589219 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" event={"ID":"cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3","Type":"ContainerDied","Data":"85b9f86242e3c88561a11548c732bea3d7ffc1a93b4464599799f7dc544e2ff9"} Oct 30 19:48:12.589894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.589288 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b9f86242e3c88561a11548c732bea3d7ffc1a93b4464599799f7dc544e2ff9" Oct 30 19:48:12.589894 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.589362 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46" Oct 30 19:48:12.852389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.852300 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:12.874511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.873976 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5jm6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:48:12.874511 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:12.874051 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:20.886678 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:48:20.886635 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:48:21.521796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:21.521757 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:48:35.104626 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.104591 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9"] Oct 30 19:48:35.105114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.105097 6514 cpu_manager.go:414] "containerMap: removing container" podUID="cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" containerName="revoke-keys" Oct 30 19:48:35.105167 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.105115 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" containerName="revoke-keys" Oct 30 19:48:35.105204 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.105180 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" containerName="revoke-keys" Oct 30 19:48:35.108282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.108262 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.114036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.113998 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:35.115098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.115077 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:35.115493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.115134 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:48:35.115493 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.115264 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:35.117161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.117140 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9"] Oct 30 19:48:35.237303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.237257 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6xj\" (UniqueName: \"kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.237500 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.237314 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.237500 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.237422 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.339483 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.339440 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6xj\" (UniqueName: \"kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.339692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.339524 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.339692 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.339588 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.340467 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.340437 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.343181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.343124 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.348649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.348624 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6xj\" (UniqueName: \"kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj\") pod \"maas-api-revoke-sub-046e1343-fgsm9\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.423132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.423045 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:35.693192 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.693164 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9"] Oct 30 19:48:35.695161 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:48:35.695133 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af16108_d165_4ec9_b3b2_15a1a539fffe.slice/crio-b28a6bd441bf3fe236763902031ebc3ea282fb89b3dd2d218b948cb80ccb47d6 WatchSource:0}: Error finding container b28a6bd441bf3fe236763902031ebc3ea282fb89b3dd2d218b948cb80ccb47d6: Status 404 returned error can't find the container with id b28a6bd441bf3fe236763902031ebc3ea282fb89b3dd2d218b948cb80ccb47d6 Oct 30 19:48:35.748571 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:35.748540 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" event={"ID":"7af16108-d165-4ec9-b3b2-15a1a539fffe","Type":"ContainerStarted","Data":"b28a6bd441bf3fe236763902031ebc3ea282fb89b3dd2d218b948cb80ccb47d6"} Oct 30 19:48:36.755022 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:36.754939 6514 generic.go:356] "Generic (PLEG): container finished" podID="7af16108-d165-4ec9-b3b2-15a1a539fffe" containerID="3d50dea0716396fc0ac9da084847f61d94606cf16ccfd9b1ee839580f193220e" exitCode=0 Oct 30 19:48:36.755408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:36.755020 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" event={"ID":"7af16108-d165-4ec9-b3b2-15a1a539fffe","Type":"ContainerDied","Data":"3d50dea0716396fc0ac9da084847f61d94606cf16ccfd9b1ee839580f193220e"} Oct 30 19:48:37.896641 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:37.896615 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:38.070457 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.070378 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca\") pod \"7af16108-d165-4ec9-b3b2-15a1a539fffe\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " Oct 30 19:48:38.070610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.070477 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca\") pod \"7af16108-d165-4ec9-b3b2-15a1a539fffe\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " Oct 30 19:48:38.070610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.070524 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj\" (UniqueName: \"kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj\") pod \"7af16108-d165-4ec9-b3b2-15a1a539fffe\" (UID: \"7af16108-d165-4ec9-b3b2-15a1a539fffe\") " Oct 30 19:48:38.070795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.070765 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca" pod "7af16108-d165-4ec9-b3b2-15a1a539fffe" (UID: "7af16108-d165-4ec9-b3b2-15a1a539fffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:38.070971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.070953 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:38.072476 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.072452 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca" pod "7af16108-d165-4ec9-b3b2-15a1a539fffe" (UID: "7af16108-d165-4ec9-b3b2-15a1a539fffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:38.072590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.072567 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj" pod "7af16108-d165-4ec9-b3b2-15a1a539fffe" (UID: "7af16108-d165-4ec9-b3b2-15a1a539fffe"). InnerVolumeSpecName "kube-api-access-zc6xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:38.171768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.171734 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af16108-d165-4ec9-b3b2-15a1a539fffe-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:38.171768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.171769 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc6xj\" (UniqueName: \"kubernetes.io/projected/7af16108-d165-4ec9-b3b2-15a1a539fffe-kube-api-access-zc6xj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:38.768695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.768662 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" Oct 30 19:48:38.768695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.768674 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9" event={"ID":"7af16108-d165-4ec9-b3b2-15a1a539fffe","Type":"ContainerDied","Data":"b28a6bd441bf3fe236763902031ebc3ea282fb89b3dd2d218b948cb80ccb47d6"} Oct 30 19:48:38.769003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:38.768717 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28a6bd441bf3fe236763902031ebc3ea282fb89b3dd2d218b948cb80ccb47d6" Oct 30 19:48:45.313227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.313187 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2"] Oct 30 19:48:45.313930 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.313905 6514 cpu_manager.go:414] "containerMap: removing container" podUID="7af16108-d165-4ec9-b3b2-15a1a539fffe" containerName="revoke-keys" Oct 30 19:48:45.314062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.313930 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af16108-d165-4ec9-b3b2-15a1a539fffe" containerName="revoke-keys" Oct 30 19:48:45.314125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.314082 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af16108-d165-4ec9-b3b2-15a1a539fffe" containerName="revoke-keys" Oct 30 19:48:45.317653 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.317633 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.324446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.324415 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:45.324569 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.324559 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:48:45.325268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.324788 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:45.325268 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.324412 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:45.325846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.325827 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2"] Oct 30 19:48:45.443835 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.443796 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.444079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.443844 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.444079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.444005 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bj5\" (UniqueName: \"kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.545174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.545145 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bj5\" (UniqueName: \"kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.545351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.545256 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.545351 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.545293 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.545916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.545887 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.547700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.547678 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.554060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.554035 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bj5\" (UniqueName: \"kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5\") pod \"maas-api-revoke-sub-d3fba056-4b9q2\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.631591 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.631540 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:45.905647 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:45.905621 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2"] Oct 30 19:48:45.907917 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:48:45.907883 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944f72f4_3b0d_42cd_915f_265c2f270bb5.slice/crio-0a586f4e10859dab767a4359555d3be6dbe76495e6d1c168d9d9af9c5a79e3c1 WatchSource:0}: Error finding container 0a586f4e10859dab767a4359555d3be6dbe76495e6d1c168d9d9af9c5a79e3c1: Status 404 returned error can't find the container with id 0a586f4e10859dab767a4359555d3be6dbe76495e6d1c168d9d9af9c5a79e3c1 Oct 30 19:48:46.026737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:46.026700 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:48:46.810788 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:46.810743 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" event={"ID":"944f72f4-3b0d-42cd-915f-265c2f270bb5","Type":"ContainerStarted","Data":"063a7cc3ad04bf05cd5fe23492d097ade52ed4e3ebeaba9194858cbad6cfa9ec"} Oct 30 19:48:46.811337 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:46.810815 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" event={"ID":"944f72f4-3b0d-42cd-915f-265c2f270bb5","Type":"ContainerStarted","Data":"0a586f4e10859dab767a4359555d3be6dbe76495e6d1c168d9d9af9c5a79e3c1"} Oct 30 19:48:46.833223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:46.833158 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" podStartSLOduration=1.8331391369999999 podStartE2EDuration="1.833139137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:46.830946394 +0000 UTC m=+1896.788765584" watchObservedRunningTime="2026-10-30 19:48:46.833139137 +0000 UTC m=+1896.790958327" Oct 30 19:48:47.872888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:47.872082 6514 generic.go:356] "Generic (PLEG): container finished" podID="944f72f4-3b0d-42cd-915f-265c2f270bb5" containerID="063a7cc3ad04bf05cd5fe23492d097ade52ed4e3ebeaba9194858cbad6cfa9ec" exitCode=0 Oct 30 19:48:47.872888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:47.872270 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" event={"ID":"944f72f4-3b0d-42cd-915f-265c2f270bb5","Type":"ContainerDied","Data":"063a7cc3ad04bf05cd5fe23492d097ade52ed4e3ebeaba9194858cbad6cfa9ec"} Oct 30 19:48:49.209634 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.209607 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:49.246016 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.245990 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca\") pod \"944f72f4-3b0d-42cd-915f-265c2f270bb5\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " Oct 30 19:48:49.246185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.246088 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca\") pod \"944f72f4-3b0d-42cd-915f-265c2f270bb5\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " Oct 30 19:48:49.246185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.246124 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5\" (UniqueName: \"kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5\") pod \"944f72f4-3b0d-42cd-915f-265c2f270bb5\" (UID: \"944f72f4-3b0d-42cd-915f-265c2f270bb5\") " Oct 30 19:48:49.246394 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.246368 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca" pod "944f72f4-3b0d-42cd-915f-265c2f270bb5" (UID: "944f72f4-3b0d-42cd-915f-265c2f270bb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:49.248187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.248157 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5" pod "944f72f4-3b0d-42cd-915f-265c2f270bb5" (UID: "944f72f4-3b0d-42cd-915f-265c2f270bb5"). InnerVolumeSpecName "kube-api-access-56bj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:49.248187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.248169 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca" pod "944f72f4-3b0d-42cd-915f-265c2f270bb5" (UID: "944f72f4-3b0d-42cd-915f-265c2f270bb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:49.347320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.347287 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:49.347320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.347315 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bj5\" (UniqueName: \"kubernetes.io/projected/944f72f4-3b0d-42cd-915f-265c2f270bb5-kube-api-access-56bj5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:49.347486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.347327 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944f72f4-3b0d-42cd-915f-265c2f270bb5-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:49.890858 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.890822 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" Oct 30 19:48:49.891062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.890842 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2" event={"ID":"944f72f4-3b0d-42cd-915f-265c2f270bb5","Type":"ContainerDied","Data":"0a586f4e10859dab767a4359555d3be6dbe76495e6d1c168d9d9af9c5a79e3c1"} Oct 30 19:48:49.891062 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:49.890922 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a586f4e10859dab767a4359555d3be6dbe76495e6d1c168d9d9af9c5a79e3c1" Oct 30 19:48:50.880854 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:48:50.880815 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:48:56.364147 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:48:56.364065 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:01.564746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:01.564698 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:02.136452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:02.136417 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:02.257041 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:02.257009 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:06.307620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.307583 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp"] Oct 30 19:49:06.308060 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.308047 6514 cpu_manager.go:414] "containerMap: removing container" podUID="944f72f4-3b0d-42cd-915f-265c2f270bb5" containerName="revoke-keys" Oct 30 19:49:06.308114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.308062 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f72f4-3b0d-42cd-915f-265c2f270bb5" containerName="revoke-keys" Oct 30 19:49:06.308218 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.308124 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="944f72f4-3b0d-42cd-915f-265c2f270bb5" containerName="revoke-keys" Oct 30 19:49:06.311110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.311074 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.319129 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.319100 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:49:06.320160 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.320140 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:49:06.320280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.320194 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:06.320280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.320204 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:06.322665 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.322644 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp"] Oct 30 19:49:06.338492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.338462 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.338600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.338516 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbhl\" (UniqueName: \"kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.338649 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.338629 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.439460 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.439420 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.439622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.439467 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbhl\" (UniqueName: \"kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.439622 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.439505 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.440175 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.440146 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.441972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.441948 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.448462 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.448438 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbhl\" (UniqueName: \"kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl\") pod \"maas-api-revoke-sub-ae594979-6jmcp\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.629086 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.629011 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:06.904310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:06.904288 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp"] Oct 30 19:49:06.906007 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:49:06.905973 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad99aef_db56_4a7d_84ba_11e5ba3d5a4e.slice/crio-f3303fe5ca136282920bf48252944ad604bd4ac007f4b0c6dfb6993311356c2c WatchSource:0}: Error finding container f3303fe5ca136282920bf48252944ad604bd4ac007f4b0c6dfb6993311356c2c: Status 404 returned error can't find the container with id f3303fe5ca136282920bf48252944ad604bd4ac007f4b0c6dfb6993311356c2c Oct 30 19:49:07.018318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:07.018287 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" event={"ID":"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e","Type":"ContainerStarted","Data":"f3303fe5ca136282920bf48252944ad604bd4ac007f4b0c6dfb6993311356c2c"} Oct 30 19:49:08.024708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:08.024628 6514 generic.go:356] "Generic (PLEG): container finished" podID="5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" containerID="ce33669ac4deaaff8439e49ca1925993d26e25f9e5d0e00ca4f239b2f93c0d35" exitCode=0 Oct 30 19:49:08.025073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:08.024707 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" event={"ID":"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e","Type":"ContainerDied","Data":"ce33669ac4deaaff8439e49ca1925993d26e25f9e5d0e00ca4f239b2f93c0d35"} Oct 30 19:49:09.168997 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.168973 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:09.268645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.268613 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl\" (UniqueName: \"kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl\") pod \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " Oct 30 19:49:09.268645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.268650 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca\") pod \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " Oct 30 19:49:09.268936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.268672 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca\") pod \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\" (UID: \"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e\") " Oct 30 19:49:09.269150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.269119 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca" pod "5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" (UID: "5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:09.271375 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.271349 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca" pod "5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" (UID: "5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:09.271482 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.271385 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl" pod "5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" (UID: "5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e"). InnerVolumeSpecName "kube-api-access-9lbhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:09.369690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.369634 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lbhl\" (UniqueName: \"kubernetes.io/projected/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-kube-api-access-9lbhl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:09.369690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.369657 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:09.369690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:09.369666 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:10.036325 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:10.036294 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" Oct 30 19:49:10.036509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:10.036336 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp" event={"ID":"5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e","Type":"ContainerDied","Data":"f3303fe5ca136282920bf48252944ad604bd4ac007f4b0c6dfb6993311356c2c"} Oct 30 19:49:10.036509 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:10.036408 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3303fe5ca136282920bf48252944ad604bd4ac007f4b0c6dfb6993311356c2c" Oct 30 19:49:12.881532 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:12.881500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:17.237419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:17.237383 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:21.096887 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:49:21.093630 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:49:21.236888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:21.217857 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:24.522013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.521982 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm"] Oct 30 19:49:24.522450 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.522435 6514 cpu_manager.go:414] "containerMap: removing container" podUID="5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" containerName="revoke-keys" Oct 30 19:49:24.522499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.522451 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" containerName="revoke-keys" Oct 30 19:49:24.522536 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.522522 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" containerName="revoke-keys" Oct 30 19:49:24.527007 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.526983 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.533180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.532569 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:24.533180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.532627 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:49:24.533180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.533136 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:49:24.533462 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.533201 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:24.536823 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.535623 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm"] Oct 30 19:49:24.683601 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.683569 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.683760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.683606 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnn4l\" (UniqueName: \"kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.683760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.683633 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.784882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.784779 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnn4l\" (UniqueName: \"kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.784882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.784829 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.785139 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.785012 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.785564 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.785537 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.787404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.787385 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.793198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.793176 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnn4l\" (UniqueName: \"kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l\") pod \"maas-api-revoke-sub-a4aef7d7-5lhfm\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:24.842732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:24.842699 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:25.120492 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:25.120458 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm"] Oct 30 19:49:25.122412 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:49:25.122384 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0250e662_d187_4797_8e21_3a08acde1921.slice/crio-60a2b40955a1cc87591a5526efd26546a57a3efcdeb744502f348d4ef2ed9649 WatchSource:0}: Error finding container 60a2b40955a1cc87591a5526efd26546a57a3efcdeb744502f348d4ef2ed9649: Status 404 returned error can't find the container with id 60a2b40955a1cc87591a5526efd26546a57a3efcdeb744502f348d4ef2ed9649 Oct 30 19:49:25.241978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:25.241941 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" event={"ID":"0250e662-d187-4797-8e21-3a08acde1921","Type":"ContainerStarted","Data":"60a2b40955a1cc87591a5526efd26546a57a3efcdeb744502f348d4ef2ed9649"} Oct 30 19:49:26.248924 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:26.248887 6514 generic.go:356] "Generic (PLEG): container finished" podID="0250e662-d187-4797-8e21-3a08acde1921" containerID="821b353e8af965bc18d378bb793163f0da7b82799eceada9be87916f2cfa4e5b" exitCode=0 Oct 30 19:49:26.249304 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:26.248963 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" event={"ID":"0250e662-d187-4797-8e21-3a08acde1921","Type":"ContainerDied","Data":"821b353e8af965bc18d378bb793163f0da7b82799eceada9be87916f2cfa4e5b"} Oct 30 19:49:27.400145 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.400104 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:27.412744 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.412718 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l\" (UniqueName: \"kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l\") pod \"0250e662-d187-4797-8e21-3a08acde1921\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " Oct 30 19:49:27.412887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.412819 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca\") pod \"0250e662-d187-4797-8e21-3a08acde1921\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " Oct 30 19:49:27.412959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.412887 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca\") pod \"0250e662-d187-4797-8e21-3a08acde1921\" (UID: \"0250e662-d187-4797-8e21-3a08acde1921\") " Oct 30 19:49:27.413175 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.413147 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca" pod "0250e662-d187-4797-8e21-3a08acde1921" (UID: "0250e662-d187-4797-8e21-3a08acde1921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:27.413261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.413241 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0250e662-d187-4797-8e21-3a08acde1921-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:27.414729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.414705 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l" pod "0250e662-d187-4797-8e21-3a08acde1921" (UID: "0250e662-d187-4797-8e21-3a08acde1921"). InnerVolumeSpecName "kube-api-access-lnn4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:27.414820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.414778 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca" pod "0250e662-d187-4797-8e21-3a08acde1921" (UID: "0250e662-d187-4797-8e21-3a08acde1921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:27.514431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.514353 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnn4l\" (UniqueName: \"kubernetes.io/projected/0250e662-d187-4797-8e21-3a08acde1921-kube-api-access-lnn4l\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:27.514431 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:27.514382 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0250e662-d187-4797-8e21-3a08acde1921-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:28.266944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:28.266908 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" event={"ID":"0250e662-d187-4797-8e21-3a08acde1921","Type":"ContainerDied","Data":"60a2b40955a1cc87591a5526efd26546a57a3efcdeb744502f348d4ef2ed9649"} Oct 30 19:49:28.266944 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:28.266932 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm" Oct 30 19:49:28.267174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:28.266961 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a2b40955a1cc87591a5526efd26546a57a3efcdeb744502f348d4ef2ed9649" Oct 30 19:49:35.462350 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:35.462313 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:40.896205 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:40.896172 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:43.495222 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.495184 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k"] Oct 30 19:49:43.495845 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.495823 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0250e662-d187-4797-8e21-3a08acde1921" containerName="revoke-keys" Oct 30 19:49:43.495978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.495848 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0250e662-d187-4797-8e21-3a08acde1921" containerName="revoke-keys" Oct 30 19:49:43.496050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.496001 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0250e662-d187-4797-8e21-3a08acde1921" containerName="revoke-keys" Oct 30 19:49:43.499347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.499324 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.506821 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.506801 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:49:43.508229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.508201 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:43.508355 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.508255 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:43.508847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.508819 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:49:43.510784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.510759 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k"] Oct 30 19:49:43.628084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.628047 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcq2k\" (UniqueName: \"kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.628245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.628099 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.628245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.628226 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.729259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.729222 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.729433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.729309 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcq2k\" (UniqueName: \"kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.729433 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.729341 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.729959 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.729931 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.731634 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.731613 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.739631 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.739602 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcq2k\" (UniqueName: \"kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k\") pod \"maas-api-revoke-sub-79805ca9-gm85k\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:43.813819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:43.813755 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:44.087012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:44.086986 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k"] Oct 30 19:49:44.089129 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:49:44.089102 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90421051_1d76_441f_97e4_b698266a20fb.slice/crio-3dd2ccacabc48ee88fcb5ec8a864cd976e40e97a1381ed24c7db2066839ab2b7 WatchSource:0}: Error finding container 3dd2ccacabc48ee88fcb5ec8a864cd976e40e97a1381ed24c7db2066839ab2b7: Status 404 returned error can't find the container with id 3dd2ccacabc48ee88fcb5ec8a864cd976e40e97a1381ed24c7db2066839ab2b7 Oct 30 19:49:44.378348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:44.378281 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" event={"ID":"90421051-1d76-441f-97e4-b698266a20fb","Type":"ContainerStarted","Data":"3dd2ccacabc48ee88fcb5ec8a864cd976e40e97a1381ed24c7db2066839ab2b7"} Oct 30 19:49:45.385320 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:45.385285 6514 generic.go:356] "Generic (PLEG): container finished" podID="90421051-1d76-441f-97e4-b698266a20fb" containerID="5592ba85acc4471cf5bf71829deb4159cfd946accf043a07ba27938f69ab80b3" exitCode=0 Oct 30 19:49:45.387707 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:45.385379 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" event={"ID":"90421051-1d76-441f-97e4-b698266a20fb","Type":"ContainerDied","Data":"5592ba85acc4471cf5bf71829deb4159cfd946accf043a07ba27938f69ab80b3"} Oct 30 19:49:46.522455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.522425 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:46.656791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.656721 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca\") pod \"90421051-1d76-441f-97e4-b698266a20fb\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " Oct 30 19:49:46.656791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.656761 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca\") pod \"90421051-1d76-441f-97e4-b698266a20fb\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " Oct 30 19:49:46.657065 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.656895 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k\" (UniqueName: \"kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k\") pod \"90421051-1d76-441f-97e4-b698266a20fb\" (UID: \"90421051-1d76-441f-97e4-b698266a20fb\") " Oct 30 19:49:46.657131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.657117 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca" pod "90421051-1d76-441f-97e4-b698266a20fb" (UID: "90421051-1d76-441f-97e4-b698266a20fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:46.657255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.657235 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90421051-1d76-441f-97e4-b698266a20fb-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:46.659011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.658988 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k" pod "90421051-1d76-441f-97e4-b698266a20fb" (UID: "90421051-1d76-441f-97e4-b698266a20fb"). InnerVolumeSpecName "kube-api-access-gcq2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:46.659150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.659127 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca" pod "90421051-1d76-441f-97e4-b698266a20fb" (UID: "90421051-1d76-441f-97e4-b698266a20fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:46.758035 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.757992 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcq2k\" (UniqueName: \"kubernetes.io/projected/90421051-1d76-441f-97e4-b698266a20fb-kube-api-access-gcq2k\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:46.758035 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:46.758037 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90421051-1d76-441f-97e4-b698266a20fb-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:47.396535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:47.396506 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" Oct 30 19:49:47.396736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:47.396539 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k" event={"ID":"90421051-1d76-441f-97e4-b698266a20fb","Type":"ContainerDied","Data":"3dd2ccacabc48ee88fcb5ec8a864cd976e40e97a1381ed24c7db2066839ab2b7"} Oct 30 19:49:47.396736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:47.396587 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd2ccacabc48ee88fcb5ec8a864cd976e40e97a1381ed24c7db2066839ab2b7" Oct 30 19:49:50.879721 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:49:50.879681 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:49:54.470108 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:54.470074 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:49:58.377846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:49:58.366509 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:01.302882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.302822 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z"] Oct 30 19:50:01.303414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.303396 6514 cpu_manager.go:414] "containerMap: removing container" podUID="90421051-1d76-441f-97e4-b698266a20fb" containerName="revoke-keys" Oct 30 19:50:01.303463 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.303416 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90421051-1d76-441f-97e4-b698266a20fb" containerName="revoke-keys" Oct 30 19:50:01.303523 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.303511 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="90421051-1d76-441f-97e4-b698266a20fb" containerName="revoke-keys" Oct 30 19:50:01.307025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.307009 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.312537 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.312511 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:01.314760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.314736 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:50:01.314896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.314830 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:50:01.314971 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.314920 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:50:01.316541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.316519 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z"] Oct 30 19:50:01.387245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.387204 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zt7\" (UniqueName: \"kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.387453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.387314 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.387453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.387374 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.487887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.487843 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zt7\" (UniqueName: \"kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.488070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.487942 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.488070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.487996 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.488721 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.488680 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.490267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.490246 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.496904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.496876 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zt7\" (UniqueName: \"kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7\") pod \"maas-api-revoke-sub-0b7e8edb-65r8z\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.621935 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.621809 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:01.896327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:01.896296 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z"] Oct 30 19:50:01.898246 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:50:01.898206 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e876c04_9ff0_4c16_9347_980143a95a57.slice/crio-8209a8ba559d7e3f25398f39ff06abdc0bea02d34a97115cfc324b36a7048d26 WatchSource:0}: Error finding container 8209a8ba559d7e3f25398f39ff06abdc0bea02d34a97115cfc324b36a7048d26: Status 404 returned error can't find the container with id 8209a8ba559d7e3f25398f39ff06abdc0bea02d34a97115cfc324b36a7048d26 Oct 30 19:50:02.484763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:02.484672 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" event={"ID":"7e876c04-9ff0-4c16-9347-980143a95a57","Type":"ContainerStarted","Data":"de64ef9339af243ba31b2dda4a8cf5ec62b2531185378602da652aa10f147247"} Oct 30 19:50:02.484763 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:02.484722 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" event={"ID":"7e876c04-9ff0-4c16-9347-980143a95a57","Type":"ContainerStarted","Data":"8209a8ba559d7e3f25398f39ff06abdc0bea02d34a97115cfc324b36a7048d26"} Oct 30 19:50:02.503716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:02.503660 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" podStartSLOduration=1.5036476109999999 podStartE2EDuration="1.503647611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:02.50160472 +0000 UTC m=+1972.459423929" watchObservedRunningTime="2026-10-30 19:50:02.503647611 +0000 UTC m=+1972.461466797" Oct 30 19:50:03.491519 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:03.491482 6514 generic.go:356] "Generic (PLEG): container finished" podID="7e876c04-9ff0-4c16-9347-980143a95a57" containerID="de64ef9339af243ba31b2dda4a8cf5ec62b2531185378602da652aa10f147247" exitCode=0 Oct 30 19:50:03.491923 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:03.491570 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" event={"ID":"7e876c04-9ff0-4c16-9347-980143a95a57","Type":"ContainerDied","Data":"de64ef9339af243ba31b2dda4a8cf5ec62b2531185378602da652aa10f147247"} Oct 30 19:50:04.631563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.631537 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:04.725941 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.725912 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7\" (UniqueName: \"kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7\") pod \"7e876c04-9ff0-4c16-9347-980143a95a57\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " Oct 30 19:50:04.726082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.725961 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca\") pod \"7e876c04-9ff0-4c16-9347-980143a95a57\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " Oct 30 19:50:04.726082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.726023 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca\") pod \"7e876c04-9ff0-4c16-9347-980143a95a57\" (UID: \"7e876c04-9ff0-4c16-9347-980143a95a57\") " Oct 30 19:50:04.726455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.726429 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca" pod "7e876c04-9ff0-4c16-9347-980143a95a57" (UID: "7e876c04-9ff0-4c16-9347-980143a95a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:04.728102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.728078 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7" pod "7e876c04-9ff0-4c16-9347-980143a95a57" (UID: "7e876c04-9ff0-4c16-9347-980143a95a57"). InnerVolumeSpecName "kube-api-access-v5zt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:04.728102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.728083 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca" pod "7e876c04-9ff0-4c16-9347-980143a95a57" (UID: "7e876c04-9ff0-4c16-9347-980143a95a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:04.826734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.826665 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:04.826734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.826695 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e876c04-9ff0-4c16-9347-980143a95a57-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:04.826734 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:04.826711 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5zt7\" (UniqueName: \"kubernetes.io/projected/7e876c04-9ff0-4c16-9347-980143a95a57-kube-api-access-v5zt7\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:05.504620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:05.504584 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" event={"ID":"7e876c04-9ff0-4c16-9347-980143a95a57","Type":"ContainerDied","Data":"8209a8ba559d7e3f25398f39ff06abdc0bea02d34a97115cfc324b36a7048d26"} Oct 30 19:50:05.504795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:05.504633 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8209a8ba559d7e3f25398f39ff06abdc0bea02d34a97115cfc324b36a7048d26" Oct 30 19:50:05.504795 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:05.504601 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z" Oct 30 19:50:12.013384 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.013349 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j"] Oct 30 19:50:12.013768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.013748 6514 cpu_manager.go:414] "containerMap: removing container" podUID="7e876c04-9ff0-4c16-9347-980143a95a57" containerName="revoke-keys" Oct 30 19:50:12.013768 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.013759 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e876c04-9ff0-4c16-9347-980143a95a57" containerName="revoke-keys" Oct 30 19:50:12.013876 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.013852 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e876c04-9ff0-4c16-9347-980143a95a57" containerName="revoke-keys" Oct 30 19:50:12.017293 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.017273 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.023273 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.023246 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:50:12.024710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.024689 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:12.024826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.024711 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:50:12.024826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.024790 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:50:12.030786 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.030761 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j"] Oct 30 19:50:12.096560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.096525 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tpb\" (UniqueName: \"kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.096733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.096592 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.096733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.096674 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.198263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.198231 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.198426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.198342 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tpb\" (UniqueName: \"kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.198426 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.198392 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.199063 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.199038 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.200667 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.200639 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.211194 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.211168 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tpb\" (UniqueName: \"kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb\") pod \"maas-api-revoke-sub-3d4156f1-h4z9j\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.255648 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.255621 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:12.331900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.331817 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:12.623758 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.623655 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j"] Oct 30 19:50:12.628853 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:50:12.628799 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe651ce8_b4bd_4f2a_b122_ddfd6f8d9c7e.slice/crio-dae5b9984d355c110028f7ae8ee79ae5f6ab17b159b9b60fed349a7accef70d5 WatchSource:0}: Error finding container dae5b9984d355c110028f7ae8ee79ae5f6ab17b159b9b60fed349a7accef70d5: Status 404 returned error can't find the container with id dae5b9984d355c110028f7ae8ee79ae5f6ab17b159b9b60fed349a7accef70d5 Oct 30 19:50:12.973059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:12.973028 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.550239 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:13.550202 6514 generic.go:356] "Generic (PLEG): container finished" podID="fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" containerID="682a708833affc13cb6f942d8f2edd618fe67dd5bf35822f40f0f8f62bb65266" exitCode=0 Oct 30 19:50:13.550654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:13.550285 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" event={"ID":"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e","Type":"ContainerDied","Data":"682a708833affc13cb6f942d8f2edd618fe67dd5bf35822f40f0f8f62bb65266"} Oct 30 19:50:13.550654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:13.550365 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" event={"ID":"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e","Type":"ContainerStarted","Data":"dae5b9984d355c110028f7ae8ee79ae5f6ab17b159b9b60fed349a7accef70d5"} Oct 30 19:50:14.691908 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.691884 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:14.826736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.826647 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca\") pod \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " Oct 30 19:50:14.826736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.826700 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca\") pod \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " Oct 30 19:50:14.827028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.826925 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb\" (UniqueName: \"kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb\") pod \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\" (UID: \"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e\") " Oct 30 19:50:14.827119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.827095 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca" pod "fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" (UID: "fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:14.827292 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.827267 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:14.828828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.828804 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb" pod "fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" (UID: "fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e"). InnerVolumeSpecName "kube-api-access-z9tpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:14.828975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.828842 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca" pod "fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" (UID: "fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:14.928886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.928830 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9tpb\" (UniqueName: \"kubernetes.io/projected/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-kube-api-access-z9tpb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:14.929070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:14.928891 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.562220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:15.562190 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" Oct 30 19:50:15.562220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:15.562215 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j" event={"ID":"fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e","Type":"ContainerDied","Data":"dae5b9984d355c110028f7ae8ee79ae5f6ab17b159b9b60fed349a7accef70d5"} Oct 30 19:50:15.562455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:15.562258 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dae5b9984d355c110028f7ae8ee79ae5f6ab17b159b9b60fed349a7accef70d5" Oct 30 19:50:17.815737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:17.815699 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:20.894083 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:50:20.894041 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:50:22.926704 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:22.926626 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:27.282669 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:27.282633 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:30.112054 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.112015 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j"] Oct 30 19:50:30.116478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.116452 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h4z9j"] Oct 30 19:50:30.137012 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.136983 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw"] Oct 30 19:50:30.137459 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.137443 6514 cpu_manager.go:414] "containerMap: removing container" podUID="fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" containerName="revoke-keys" Oct 30 19:50:30.137513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.137461 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" containerName="revoke-keys" Oct 30 19:50:30.137560 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.137547 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" containerName="revoke-keys" Oct 30 19:50:30.141134 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.141116 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.145847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.145807 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:30.146663 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.146633 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:50:30.146782 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.146667 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:50:30.147146 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.147127 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:50:30.149269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.149204 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw"] Oct 30 19:50:30.165711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.165677 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e" path="/var/lib/kubelet/pods/fe651ce8-b4bd-4f2a-b122-ddfd6f8d9c7e/volumes" Oct 30 19:50:30.184897 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.184850 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47pr\" (UniqueName: \"kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.185048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.185000 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.185048 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.185034 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.285661 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.285625 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.285806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.285675 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.285806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.285753 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47pr\" (UniqueName: \"kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.286466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.286440 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.287943 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.287922 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.295274 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.295250 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47pr\" (UniqueName: \"kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr\") pod \"maas-api-revoke-sub-3d4156f1-pf2bw\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.456690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.456669 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:30.728748 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:30.728708 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw"] Oct 30 19:50:30.729800 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:50:30.729762 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fe4ac3_923c_4229_8dd4_af580861c637.slice/crio-336c61a5df6daa58810fe4b1093dda555eb907d3da5e972a9ba66355ae0c21ce WatchSource:0}: Error finding container 336c61a5df6daa58810fe4b1093dda555eb907d3da5e972a9ba66355ae0c21ce: Status 404 returned error can't find the container with id 336c61a5df6daa58810fe4b1093dda555eb907d3da5e972a9ba66355ae0c21ce Oct 30 19:50:31.162671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:31.162639 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.693965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:31.693932 6514 generic.go:356] "Generic (PLEG): container finished" podID="57fe4ac3-923c-4229-8dd4-af580861c637" containerID="ccc3879ba22bcf5cc6d77f907a5caa8273268b6e7fa93ab31725ef7e8307e5ef" exitCode=0 Oct 30 19:50:31.694120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:31.694015 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" event={"ID":"57fe4ac3-923c-4229-8dd4-af580861c637","Type":"ContainerDied","Data":"ccc3879ba22bcf5cc6d77f907a5caa8273268b6e7fa93ab31725ef7e8307e5ef"} Oct 30 19:50:31.694120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:31.694072 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" event={"ID":"57fe4ac3-923c-4229-8dd4-af580861c637","Type":"ContainerStarted","Data":"336c61a5df6daa58810fe4b1093dda555eb907d3da5e972a9ba66355ae0c21ce"} Oct 30 19:50:32.831399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.831375 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:32.909013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.908987 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr\" (UniqueName: \"kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr\") pod \"57fe4ac3-923c-4229-8dd4-af580861c637\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " Oct 30 19:50:32.909190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.909024 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca\") pod \"57fe4ac3-923c-4229-8dd4-af580861c637\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " Oct 30 19:50:32.909190 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.909063 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca\") pod \"57fe4ac3-923c-4229-8dd4-af580861c637\" (UID: \"57fe4ac3-923c-4229-8dd4-af580861c637\") " Oct 30 19:50:32.909408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.909386 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca" pod "57fe4ac3-923c-4229-8dd4-af580861c637" (UID: "57fe4ac3-923c-4229-8dd4-af580861c637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:32.911166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.911141 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca" pod "57fe4ac3-923c-4229-8dd4-af580861c637" (UID: "57fe4ac3-923c-4229-8dd4-af580861c637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:32.911284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:32.911219 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr" pod "57fe4ac3-923c-4229-8dd4-af580861c637" (UID: "57fe4ac3-923c-4229-8dd4-af580861c637"). InnerVolumeSpecName "kube-api-access-g47pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:33.009914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:33.009830 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g47pr\" (UniqueName: \"kubernetes.io/projected/57fe4ac3-923c-4229-8dd4-af580861c637-kube-api-access-g47pr\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:33.009914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:33.009854 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:33.009914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:33.009887 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fe4ac3-923c-4229-8dd4-af580861c637-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:33.704967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:33.704935 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" Oct 30 19:50:33.705136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:33.704967 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw" event={"ID":"57fe4ac3-923c-4229-8dd4-af580861c637","Type":"ContainerDied","Data":"336c61a5df6daa58810fe4b1093dda555eb907d3da5e972a9ba66355ae0c21ce"} Oct 30 19:50:33.705136 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:33.705014 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336c61a5df6daa58810fe4b1093dda555eb907d3da5e972a9ba66355ae0c21ce" Oct 30 19:50:41.366755 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.366718 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz"] Oct 30 19:50:41.367265 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.366910 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" podUID="15316da0-3a8c-430d-a60b-dfa38e574a42" containerName="discovery" containerID="cri-o://97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511" gracePeriod=30 Oct 30 19:50:41.620981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.620925 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:50:41.696184 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696152 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696191 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696234 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696270 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696309 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696347 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696343 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696460 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs\") pod \"15316da0-3a8c-430d-a60b-dfa38e574a42\" (UID: \"15316da0-3a8c-430d-a60b-dfa38e574a42\") " Oct 30 19:50:41.696732 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696702 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:41.696912 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.696858 6514 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-ca-configmap\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.698597 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.698559 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:41.698725 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.698610 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "kube-api-access-68tkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:41.698810 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.698790 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:41.698919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.698898 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:41.699033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.699015 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:41.699110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.699050 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token" pod "15316da0-3a8c-430d-a60b-dfa38e574a42" (UID: "15316da0-3a8c-430d-a60b-dfa38e574a42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:41.747952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.747923 6514 generic.go:356] "Generic (PLEG): container finished" podID="15316da0-3a8c-430d-a60b-dfa38e574a42" containerID="97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511" exitCode=0 Oct 30 19:50:41.748084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.747970 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" Oct 30 19:50:41.748084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.748041 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" event={"ID":"15316da0-3a8c-430d-a60b-dfa38e574a42","Type":"ContainerDied","Data":"97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511"} Oct 30 19:50:41.748302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.748100 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz" event={"ID":"15316da0-3a8c-430d-a60b-dfa38e574a42","Type":"ContainerDied","Data":"cbc0935825fe7ee3755c0c8ea7929240c3e966fe2baa9ab67c965be13f1f2095"} Oct 30 19:50:41.748302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.748128 6514 scope.go:122] "RemoveContainer" containerID="97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511" Oct 30 19:50:41.764992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.764970 6514 scope.go:122] "RemoveContainer" containerID="97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511" Oct 30 19:50:41.765248 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:50:41.765229 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511\": container with ID starting with 97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511 not found: ID does not exist" containerID="97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511" Oct 30 19:50:41.765287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.765257 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511"} err="failed to get container status \"97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511\": rpc error: code = NotFound desc = could not find container \"97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511\": container with ID starting with 97f3596c5fb65f96698f3e7318fd2f8c2e31e4ddaff0c2257c5a9160c411c511 not found: ID does not exist" Oct 30 19:50:41.772373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.772352 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz"] Oct 30 19:50:41.776478 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.776458 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mfsgz"] Oct 30 19:50:41.797846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.797822 6514 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15316da0-3a8c-430d-a60b-dfa38e574a42-local-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.797846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.797847 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.797999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.797858 6514 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-kubeconfig\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.797999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.797893 6514 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-istio-csr-dns-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.797999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.797903 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68tkv\" (UniqueName: \"kubernetes.io/projected/15316da0-3a8c-430d-a60b-dfa38e574a42-kube-api-access-68tkv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.797999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.797912 6514 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15316da0-3a8c-430d-a60b-dfa38e574a42-cacerts\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:41.967230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:41.967201 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:42.158594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:42.158561 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15316da0-3a8c-430d-a60b-dfa38e574a42" path="/var/lib/kubelet/pods/15316da0-3a8c-430d-a60b-dfa38e574a42/volumes" Oct 30 19:50:42.600832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:42.600799 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.751291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:45.751255 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:50.887468 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:50:50.887420 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:50:54.462229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:54.462193 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:50:57.014887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:56.991946 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s6ldn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:50:57.014887 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:56.992030 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:57.037024 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:57.036988 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-742kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:50:57.037344 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:57.037292 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:57.436884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:57.435954 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-lnkcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:50:57.436884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:50:57.436093 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:01.936956 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.936918 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj"] Oct 30 19:51:01.937510 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.937489 6514 cpu_manager.go:414] "containerMap: removing container" podUID="15316da0-3a8c-430d-a60b-dfa38e574a42" containerName="discovery" Oct 30 19:51:01.937575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.937511 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15316da0-3a8c-430d-a60b-dfa38e574a42" containerName="discovery" Oct 30 19:51:01.937575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.937524 6514 cpu_manager.go:414] "containerMap: removing container" podUID="57fe4ac3-923c-4229-8dd4-af580861c637" containerName="revoke-keys" Oct 30 19:51:01.937575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.937530 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fe4ac3-923c-4229-8dd4-af580861c637" containerName="revoke-keys" Oct 30 19:51:01.937673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.937611 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="15316da0-3a8c-430d-a60b-dfa38e574a42" containerName="discovery" Oct 30 19:51:01.937673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.937620 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fe4ac3-923c-4229-8dd4-af580861c637" containerName="revoke-keys" Oct 30 19:51:01.942103 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.942087 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:01.945978 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.945949 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:51:01.947264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.947242 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:01.947506 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.947490 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:51:01.947714 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.947678 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:51:01.951339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:01.950599 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj"] Oct 30 19:51:02.074617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.074580 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.074796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.074631 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.074796 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.074736 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltfx\" (UniqueName: \"kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.175303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.175270 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.175494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.175321 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltfx\" (UniqueName: \"kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.175494 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.175390 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.176078 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.176053 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.177642 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.177618 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.183833 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.183810 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltfx\" (UniqueName: \"kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx\") pod \"maas-api-revoke-sub-e2fc9c5b-685nj\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.259808 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.259730 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:02.532209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.532185 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj"] Oct 30 19:51:02.533847 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:51:02.533812 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bceed4_7d90_4ffc_98de_e459f38feffd.slice/crio-06a66b3fe01d375320e15294e1530d19bf3fefc4355af15f2c2a7a8b289d6cfe WatchSource:0}: Error finding container 06a66b3fe01d375320e15294e1530d19bf3fefc4355af15f2c2a7a8b289d6cfe: Status 404 returned error can't find the container with id 06a66b3fe01d375320e15294e1530d19bf3fefc4355af15f2c2a7a8b289d6cfe Oct 30 19:51:02.893926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.893892 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" event={"ID":"b4bceed4-7d90-4ffc-98de-e459f38feffd","Type":"ContainerStarted","Data":"852dde386b3d81493500d288d2c4d1e244d94f05b7b47efcede934d1df1f7f8c"} Oct 30 19:51:02.894138 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.893953 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" event={"ID":"b4bceed4-7d90-4ffc-98de-e459f38feffd","Type":"ContainerStarted","Data":"06a66b3fe01d375320e15294e1530d19bf3fefc4355af15f2c2a7a8b289d6cfe"} Oct 30 19:51:02.912832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:02.912779 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" podStartSLOduration=1.912764498 podStartE2EDuration="1.912764498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:02.911725737 +0000 UTC m=+2032.869544924" watchObservedRunningTime="2026-10-30 19:51:02.912764498 +0000 UTC m=+2032.870583687" Oct 30 19:51:03.900856 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:03.900815 6514 generic.go:356] "Generic (PLEG): container finished" podID="b4bceed4-7d90-4ffc-98de-e459f38feffd" containerID="852dde386b3d81493500d288d2c4d1e244d94f05b7b47efcede934d1df1f7f8c" exitCode=0 Oct 30 19:51:03.901372 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:03.900895 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" event={"ID":"b4bceed4-7d90-4ffc-98de-e459f38feffd","Type":"ContainerDied","Data":"852dde386b3d81493500d288d2c4d1e244d94f05b7b47efcede934d1df1f7f8c"} Oct 30 19:51:05.042486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.042455 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:05.104690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.104653 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca\") pod \"b4bceed4-7d90-4ffc-98de-e459f38feffd\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " Oct 30 19:51:05.104927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.104708 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx\" (UniqueName: \"kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx\") pod \"b4bceed4-7d90-4ffc-98de-e459f38feffd\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " Oct 30 19:51:05.104927 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.104788 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca\") pod \"b4bceed4-7d90-4ffc-98de-e459f38feffd\" (UID: \"b4bceed4-7d90-4ffc-98de-e459f38feffd\") " Oct 30 19:51:05.105174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.105148 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca" pod "b4bceed4-7d90-4ffc-98de-e459f38feffd" (UID: "b4bceed4-7d90-4ffc-98de-e459f38feffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:05.106883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.106843 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx" pod "b4bceed4-7d90-4ffc-98de-e459f38feffd" (UID: "b4bceed4-7d90-4ffc-98de-e459f38feffd"). InnerVolumeSpecName "kube-api-access-4ltfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:05.106969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.106927 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca" pod "b4bceed4-7d90-4ffc-98de-e459f38feffd" (UID: "b4bceed4-7d90-4ffc-98de-e459f38feffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:05.206034 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.206000 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:05.206220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.206207 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltfx\" (UniqueName: \"kubernetes.io/projected/b4bceed4-7d90-4ffc-98de-e459f38feffd-kube-api-access-4ltfx\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:05.206289 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.206229 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4bceed4-7d90-4ffc-98de-e459f38feffd-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:05.913096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.913063 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" Oct 30 19:51:05.913096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.913071 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj" event={"ID":"b4bceed4-7d90-4ffc-98de-e459f38feffd","Type":"ContainerDied","Data":"06a66b3fe01d375320e15294e1530d19bf3fefc4355af15f2c2a7a8b289d6cfe"} Oct 30 19:51:05.913318 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:05.913121 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a66b3fe01d375320e15294e1530d19bf3fefc4355af15f2c2a7a8b289d6cfe" Oct 30 19:51:12.143761 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.143726 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2"] Oct 30 19:51:12.144303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.144195 6514 cpu_manager.go:414] "containerMap: removing container" podUID="b4bceed4-7d90-4ffc-98de-e459f38feffd" containerName="revoke-keys" Oct 30 19:51:12.144303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.144209 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bceed4-7d90-4ffc-98de-e459f38feffd" containerName="revoke-keys" Oct 30 19:51:12.144303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.144283 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bceed4-7d90-4ffc-98de-e459f38feffd" containerName="revoke-keys" Oct 30 19:51:12.151185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.151158 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.157167 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.157105 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:51:12.157343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.157256 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:51:12.157414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.157354 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:51:12.157414 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.157364 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:12.162662 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.162632 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2"] Oct 30 19:51:12.278004 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.277964 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.278199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.278145 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.278284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.278201 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdxt\" (UniqueName: \"kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.379330 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.379292 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.379535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.379368 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.379535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.379490 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdxt\" (UniqueName: \"kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.379965 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.379944 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.381693 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.381670 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.387716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.387696 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdxt\" (UniqueName: \"kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt\") pod \"maas-api-revoke-sub-5a29efb3-7nxl2\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.473069 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.473037 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:12.751025 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.750994 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2"] Oct 30 19:51:12.752486 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:51:12.752447 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421b1746_c786_4ad7_ba5d_1d70f175db31.slice/crio-df67b7c357c849d5b3dfeb87b994f01b89f80a089bbd79adfc93f82f4407c945 WatchSource:0}: Error finding container df67b7c357c849d5b3dfeb87b994f01b89f80a089bbd79adfc93f82f4407c945: Status 404 returned error can't find the container with id df67b7c357c849d5b3dfeb87b994f01b89f80a089bbd79adfc93f82f4407c945 Oct 30 19:51:12.821100 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.821070 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:51:12.953073 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.953038 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" event={"ID":"421b1746-c786-4ad7-ba5d-1d70f175db31","Type":"ContainerStarted","Data":"df67b7c357c849d5b3dfeb87b994f01b89f80a089bbd79adfc93f82f4407c945"} Oct 30 19:51:12.988404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:12.988373 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:13.963746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:13.963646 6514 generic.go:356] "Generic (PLEG): container finished" podID="421b1746-c786-4ad7-ba5d-1d70f175db31" containerID="56769e3f6f913391ba7c8fb766c5cf467b917d4f4ed2475d69a4334f7112a9a5" exitCode=0 Oct 30 19:51:13.963746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:13.963707 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" event={"ID":"421b1746-c786-4ad7-ba5d-1d70f175db31","Type":"ContainerDied","Data":"56769e3f6f913391ba7c8fb766c5cf467b917d4f4ed2475d69a4334f7112a9a5"} Oct 30 19:51:15.608276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.608240 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:15.684251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.684212 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca\") pod \"421b1746-c786-4ad7-ba5d-1d70f175db31\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " Oct 30 19:51:15.689306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.689275 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt\" (UniqueName: \"kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt\") pod \"421b1746-c786-4ad7-ba5d-1d70f175db31\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " Oct 30 19:51:15.694883 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.689851 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca\") pod \"421b1746-c786-4ad7-ba5d-1d70f175db31\" (UID: \"421b1746-c786-4ad7-ba5d-1d70f175db31\") " Oct 30 19:51:15.700914 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.689070 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca" pod "421b1746-c786-4ad7-ba5d-1d70f175db31" (UID: "421b1746-c786-4ad7-ba5d-1d70f175db31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:15.705885 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.701534 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:15.715185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.715146 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt" pod "421b1746-c786-4ad7-ba5d-1d70f175db31" (UID: "421b1746-c786-4ad7-ba5d-1d70f175db31"). InnerVolumeSpecName "kube-api-access-pvdxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:15.742412 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.742373 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca" pod "421b1746-c786-4ad7-ba5d-1d70f175db31" (UID: "421b1746-c786-4ad7-ba5d-1d70f175db31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:15.825945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.824086 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdxt\" (UniqueName: \"kubernetes.io/projected/421b1746-c786-4ad7-ba5d-1d70f175db31-kube-api-access-pvdxt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:15.825945 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.824137 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b1746-c786-4ad7-ba5d-1d70f175db31-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:15.987080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.987046 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" Oct 30 19:51:15.987080 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.987069 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2" event={"ID":"421b1746-c786-4ad7-ba5d-1d70f175db31","Type":"ContainerDied","Data":"df67b7c357c849d5b3dfeb87b994f01b89f80a089bbd79adfc93f82f4407c945"} Oct 30 19:51:15.987300 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:15.987131 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df67b7c357c849d5b3dfeb87b994f01b89f80a089bbd79adfc93f82f4407c945" Oct 30 19:51:19.930822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:19.930790 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.882100 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:51:20.882057 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:51:23.071243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:23.071211 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:51:49.025084 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:49.025048 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5"] Oct 30 19:51:49.027877 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:49.027836 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xx5r5"] Oct 30 19:51:50.157791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:51:50.157759 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4556a91f-9de2-4309-b3b5-dfbb749dd8b2" path="/var/lib/kubelet/pods/4556a91f-9de2-4309-b3b5-dfbb749dd8b2/volumes" Oct 30 19:51:50.881155 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:51:50.881111 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:52:10.741774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:10.741688 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:52:10.746608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:10.746587 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:52:13.029847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:13.029810 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:13.132343 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:13.132312 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.883174 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:52:20.883131 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:52:43.033710 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:43.033673 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8"] Oct 30 19:52:43.035514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:43.035488 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ngms8"] Oct 30 19:52:44.157353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:52:44.157320 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abbb50f-b8d8-40d1-b1bf-043ffd2c255c" path="/var/lib/kubelet/pods/4abbb50f-b8d8-40d1-b1bf-043ffd2c255c/volumes" Oct 30 19:52:50.881048 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:52:50.881015 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:53:01.024381 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:01.024346 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t"] Oct 30 19:53:01.027361 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:01.027330 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x5w5t"] Oct 30 19:53:02.158694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:02.158655 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a6ee8c-7b77-4b54-96e2-ce8b84068a87" path="/var/lib/kubelet/pods/b3a6ee8c-7b77-4b54-96e2-ce8b84068a87/volumes" Oct 30 19:53:12.028776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:12.028735 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46"] Oct 30 19:53:12.033596 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:12.033568 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-fgj46"] Oct 30 19:53:12.157826 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:12.157784 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3" path="/var/lib/kubelet/pods/cdc8c4c1-61f1-443d-95c2-d1c694fe6bc3/volumes" Oct 30 19:53:13.008794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:13.008763 6514 scope.go:122] "RemoveContainer" containerID="c7b6a87b6072bc74f71a2e75e21d47a13cd189a15072b1b013cec1b6d11731e4" Oct 30 19:53:13.091033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:13.090997 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:13.206294 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:13.206264 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:20.884070 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:53:20.884033 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:53:38.033340 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:38.033260 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9"] Oct 30 19:53:38.036938 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:38.036913 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fgsm9"] Oct 30 19:53:38.157225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:38.157183 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af16108-d165-4ec9-b3b2-15a1a539fffe" path="/var/lib/kubelet/pods/7af16108-d165-4ec9-b3b2-15a1a539fffe/volumes" Oct 30 19:53:49.024485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:49.024445 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2"] Oct 30 19:53:49.026779 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:49.026753 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4b9q2"] Oct 30 19:53:50.157739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:53:50.157706 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944f72f4-3b0d-42cd-915f-265c2f270bb5" path="/var/lib/kubelet/pods/944f72f4-3b0d-42cd-915f-265c2f270bb5/volumes" Oct 30 19:53:50.887731 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:53:50.887691 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:54:10.022948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:10.022914 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp"] Oct 30 19:54:10.025852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:10.025826 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6jmcp"] Oct 30 19:54:10.157590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:10.157554 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e" path="/var/lib/kubelet/pods/5ad99aef-db56-4a7d-84ba-11e5ba3d5a4e/volumes" Oct 30 19:54:13.030928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:13.030893 6514 scope.go:122] "RemoveContainer" containerID="9051eeb1c4b34578c078690de58c1683e01a2d29dfa172aea4560e6d20110c99" Oct 30 19:54:13.039830 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:13.039811 6514 scope.go:122] "RemoveContainer" containerID="f0e99493ddc43e8d9adb4d07e0910f16df659805b14727f12e4a420bba19d0ad" Oct 30 19:54:13.048044 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:13.048025 6514 scope.go:122] "RemoveContainer" containerID="0c6d0563958ce7131e596d503997217fa2a6866b81ceacb4a0e7ab78b03895dd" Oct 30 19:54:13.134284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:13.134252 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:20.883098 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:54:20.883057 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:54:28.023224 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:28.023193 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm"] Oct 30 19:54:28.026046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:28.026020 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5lhfm"] Oct 30 19:54:28.157033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:28.157001 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0250e662-d187-4797-8e21-3a08acde1921" path="/var/lib/kubelet/pods/0250e662-d187-4797-8e21-3a08acde1921/volumes" Oct 30 19:54:47.022121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:47.022087 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k"] Oct 30 19:54:47.024694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:47.024329 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gm85k"] Oct 30 19:54:48.157479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:54:48.157444 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90421051-1d76-441f-97e4-b698266a20fb" path="/var/lib/kubelet/pods/90421051-1d76-441f-97e4-b698266a20fb/volumes" Oct 30 19:54:50.884092 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:54:50.884047 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:55:05.026360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:05.026272 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z"] Oct 30 19:55:05.029675 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:05.029645 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-65r8z"] Oct 30 19:55:06.157299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:06.157266 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e876c04-9ff0-4c16-9347-980143a95a57" path="/var/lib/kubelet/pods/7e876c04-9ff0-4c16-9347-980143a95a57/volumes" Oct 30 19:55:13.079465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:13.079430 6514 scope.go:122] "RemoveContainer" containerID="3d50dea0716396fc0ac9da084847f61d94606cf16ccfd9b1ee839580f193220e" Oct 30 19:55:13.088128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:13.088111 6514 scope.go:122] "RemoveContainer" containerID="ce33669ac4deaaff8439e49ca1925993d26e25f9e5d0e00ca4f239b2f93c0d35" Oct 30 19:55:13.096015 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:13.095994 6514 scope.go:122] "RemoveContainer" containerID="063a7cc3ad04bf05cd5fe23492d097ade52ed4e3ebeaba9194858cbad6cfa9ec" Oct 30 19:55:13.169202 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:13.169173 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:20.882045 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:55:20.882009 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:55:33.022846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:33.022808 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw"] Oct 30 19:55:33.025208 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:33.025182 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pf2bw"] Oct 30 19:55:34.156957 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:34.156924 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fe4ac3-923c-4229-8dd4-af580861c637" path="/var/lib/kubelet/pods/57fe4ac3-923c-4229-8dd4-af580861c637/volumes" Oct 30 19:55:48.119640 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:55:48.119585 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:50.879781 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:55:50.879738 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:56:05.024082 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:05.024042 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj"] Oct 30 19:56:05.026630 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:05.026606 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-685nj"] Oct 30 19:56:06.157972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:06.157938 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bceed4-7d90-4ffc-98de-e459f38feffd" path="/var/lib/kubelet/pods/b4bceed4-7d90-4ffc-98de-e459f38feffd/volumes" Oct 30 19:56:07.488794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:07.488757 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.123556 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:13.123524 6514 scope.go:122] "RemoveContainer" containerID="5592ba85acc4471cf5bf71829deb4159cfd946accf043a07ba27938f69ab80b3" Oct 30 19:56:13.134851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:13.134805 6514 scope.go:122] "RemoveContainer" containerID="682a708833affc13cb6f942d8f2edd618fe67dd5bf35822f40f0f8f62bb65266" Oct 30 19:56:13.142895 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:13.142874 6514 scope.go:122] "RemoveContainer" containerID="821b353e8af965bc18d378bb793163f0da7b82799eceada9be87916f2cfa4e5b" Oct 30 19:56:13.150287 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:13.150270 6514 scope.go:122] "RemoveContainer" containerID="de64ef9339af243ba31b2dda4a8cf5ec62b2531185378602da652aa10f147247" Oct 30 19:56:13.220391 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:13.220363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.025261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:16.025223 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2"] Oct 30 19:56:16.028093 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:16.028068 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7nxl2"] Oct 30 19:56:16.157323 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:16.157291 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421b1746-c786-4ad7-ba5d-1d70f175db31" path="/var/lib/kubelet/pods/421b1746-c786-4ad7-ba5d-1d70f175db31/volumes" Oct 30 19:56:20.883851 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:56:20.883809 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:56:27.565953 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.565854 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl"] Oct 30 19:56:27.566417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.566306 6514 cpu_manager.go:414] "containerMap: removing container" podUID="421b1746-c786-4ad7-ba5d-1d70f175db31" containerName="revoke-keys" Oct 30 19:56:27.566417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.566317 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421b1746-c786-4ad7-ba5d-1d70f175db31" containerName="revoke-keys" Oct 30 19:56:27.566417 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.566391 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="421b1746-c786-4ad7-ba5d-1d70f175db31" containerName="revoke-keys" Oct 30 19:56:27.571324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.571302 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.575609 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.575517 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:56:27.576853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.576831 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:27.576853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.576831 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:56:27.577045 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.576840 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:56:27.582995 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.582974 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl"] Oct 30 19:56:27.651514 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.651484 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.651644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.651524 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfss\" (UniqueName: \"kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.651644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.651621 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.752611 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.752583 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.752730 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.752671 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.752852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.752830 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfss\" (UniqueName: \"kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.753370 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.753348 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.755009 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.754983 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.761126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.761105 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfss\" (UniqueName: \"kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss\") pod \"maas-api-revoke-sub-3d4156f1-v8vwl\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:27.885900 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:27.885811 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:28.160266 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:28.160240 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl"] Oct 30 19:56:28.162781 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:56:28.162750 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc054dead_d221_4741_8ea4_d79d1204158f.slice/crio-f6b36f52a4e643fd623491d9844a90d8153616f1d70c7beb1766d2c78b97c66e WatchSource:0}: Error finding container f6b36f52a4e643fd623491d9844a90d8153616f1d70c7beb1766d2c78b97c66e: Status 404 returned error can't find the container with id f6b36f52a4e643fd623491d9844a90d8153616f1d70c7beb1766d2c78b97c66e Oct 30 19:56:28.560515 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:28.560480 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" event={"ID":"c054dead-d221-4741-8ea4-d79d1204158f","Type":"ContainerStarted","Data":"2d0bf3d688e4649d58ec508be48a49af904b34084bc3a13a8e7df8a5fdbcea1c"} Oct 30 19:56:28.560671 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:28.560544 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" event={"ID":"c054dead-d221-4741-8ea4-d79d1204158f","Type":"ContainerStarted","Data":"f6b36f52a4e643fd623491d9844a90d8153616f1d70c7beb1766d2c78b97c66e"} Oct 30 19:56:28.580528 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:28.580483 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" podStartSLOduration=1.580470702 podStartE2EDuration="1.580470702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:28.578224303 +0000 UTC m=+2358.536043491" watchObservedRunningTime="2026-10-30 19:56:28.580470702 +0000 UTC m=+2358.538289902" Oct 30 19:56:29.566529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:29.566491 6514 generic.go:356] "Generic (PLEG): container finished" podID="c054dead-d221-4741-8ea4-d79d1204158f" containerID="2d0bf3d688e4649d58ec508be48a49af904b34084bc3a13a8e7df8a5fdbcea1c" exitCode=0 Oct 30 19:56:29.566715 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:29.566577 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" event={"ID":"c054dead-d221-4741-8ea4-d79d1204158f","Type":"ContainerDied","Data":"2d0bf3d688e4649d58ec508be48a49af904b34084bc3a13a8e7df8a5fdbcea1c"} Oct 30 19:56:30.707126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.707101 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:30.782525 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.782495 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca\") pod \"c054dead-d221-4741-8ea4-d79d1204158f\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " Oct 30 19:56:30.782670 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.782595 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca\") pod \"c054dead-d221-4741-8ea4-d79d1204158f\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " Oct 30 19:56:30.782733 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.782671 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss\" (UniqueName: \"kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss\") pod \"c054dead-d221-4741-8ea4-d79d1204158f\" (UID: \"c054dead-d221-4741-8ea4-d79d1204158f\") " Oct 30 19:56:30.782852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.782825 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca" pod "c054dead-d221-4741-8ea4-d79d1204158f" (UID: "c054dead-d221-4741-8ea4-d79d1204158f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:30.782975 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.782950 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:30.784702 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.784677 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca" pod "c054dead-d221-4741-8ea4-d79d1204158f" (UID: "c054dead-d221-4741-8ea4-d79d1204158f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:30.784790 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.784678 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss" pod "c054dead-d221-4741-8ea4-d79d1204158f" (UID: "c054dead-d221-4741-8ea4-d79d1204158f"). InnerVolumeSpecName "kube-api-access-lzfss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:30.883834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.883778 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c054dead-d221-4741-8ea4-d79d1204158f-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:30.883834 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:30.883803 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfss\" (UniqueName: \"kubernetes.io/projected/c054dead-d221-4741-8ea4-d79d1204158f-kube-api-access-lzfss\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:31.578021 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:31.577983 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" event={"ID":"c054dead-d221-4741-8ea4-d79d1204158f","Type":"ContainerDied","Data":"f6b36f52a4e643fd623491d9844a90d8153616f1d70c7beb1766d2c78b97c66e"} Oct 30 19:56:31.578196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:31.578039 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b36f52a4e643fd623491d9844a90d8153616f1d70c7beb1766d2c78b97c66e" Oct 30 19:56:31.578196 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:31.578000 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl" Oct 30 19:56:38.452002 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:38.451950 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:56:42.338352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:42.338320 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:56:45.828280 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:45.828244 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.463832 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.463795 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr"] Oct 30 19:56:46.464230 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.464216 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c054dead-d221-4741-8ea4-d79d1204158f" containerName="revoke-keys" Oct 30 19:56:46.464297 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.464231 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c054dead-d221-4741-8ea4-d79d1204158f" containerName="revoke-keys" Oct 30 19:56:46.464339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.464321 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c054dead-d221-4741-8ea4-d79d1204158f" containerName="revoke-keys" Oct 30 19:56:46.467602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.467583 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.473739 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.473715 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-98b568-openshift-default-dockercfg-d7tm5\"" Oct 30 19:56:46.493354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.493326 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr"] Oct 30 19:56:46.581520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581471 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581527 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfwp\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581717 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581616 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581717 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581761 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581811 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581782 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581817 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.581952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.581928 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.582030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.582016 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.682939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.682858 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.682975 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683014 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfwp\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683051 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683113 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683147 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683179 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683216 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683448 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683476 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683577 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683514 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683578 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.683708 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.683615 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.684099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.684072 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.685327 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.685308 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.685774 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.685753 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.692221 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.692194 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfwp\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.692485 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.692465 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token\") pod \"e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.780379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.780282 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:46.827199 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:46.827163 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:47.061884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.061844 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr"] Oct 30 19:56:47.064208 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:56:47.064182 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde773b7_2d55_436d_97f5_dbe701edf49a.slice/crio-cd1b826a68e69b65b3dec460ba65e6c520c1db38c0f0e121b6cff1fcb2d673c9 WatchSource:0}: Error finding container cd1b826a68e69b65b3dec460ba65e6c520c1db38c0f0e121b6cff1fcb2d673c9: Status 404 returned error can't find the container with id cd1b826a68e69b65b3dec460ba65e6c520c1db38c0f0e121b6cff1fcb2d673c9 Oct 30 19:56:47.066405 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.066371 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:47.066518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.066435 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:47.066518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.066462 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:56:47.289996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.289926 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv"] Oct 30 19:56:47.290223 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.290196 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" containerID="cri-o://0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a" gracePeriod=30 Oct 30 19:56:47.291283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.291261 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf"] Oct 30 19:56:47.291475 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.291455 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" containerID="cri-o://d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd" gracePeriod=30 Oct 30 19:56:47.292529 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.292502 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn"] Oct 30 19:56:47.292729 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.292705 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" containerID="cri-o://0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529" gracePeriod=30 Oct 30 19:56:47.294452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.294019 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z"] Oct 30 19:56:47.294452 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.294335 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" containerID="cri-o://004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52" gracePeriod=30 Oct 30 19:56:47.669003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.668960 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" event={"ID":"dde773b7-2d55-436d-97f5-dbe701edf49a","Type":"ContainerStarted","Data":"a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e"} Oct 30 19:56:47.669229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.669034 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" event={"ID":"dde773b7-2d55-436d-97f5-dbe701edf49a","Type":"ContainerStarted","Data":"cd1b826a68e69b65b3dec460ba65e6c520c1db38c0f0e121b6cff1fcb2d673c9"} Oct 30 19:56:47.692157 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.692109 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podStartSLOduration=1.692095579 podStartE2EDuration="1.692095579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:47.691139697 +0000 UTC m=+2377.648958885" watchObservedRunningTime="2026-10-30 19:56:47.692095579 +0000 UTC m=+2377.649914765" Oct 30 19:56:47.780913 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:47.780858 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:48.786453 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:48.786418 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:49.678598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:49.678564 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:49.679606 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:49.679589 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:56:50.119674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:50.119592 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.227672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:50.227643 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.880480 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:56:50.880442 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:56:51.178896 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.178823 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8"] Oct 30 19:56:51.183557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.183531 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.191855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.191417 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:56:51.191855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.191589 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2z7lq\"" Oct 30 19:56:51.191855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.191638 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-98b568-serving-cert\"" Oct 30 19:56:51.191855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.191725 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:56:51.191855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.191819 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:56:51.192854 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.192746 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8"] Oct 30 19:56:51.230540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.230501 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.230736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.230707 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.230891 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.230853 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.230949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.230918 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbq2\" (UniqueName: \"kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.265489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.265435 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.311389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.311355 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd"] Oct 30 19:56:51.315029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.315006 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.323133 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.323102 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-98b568\"" Oct 30 19:56:51.323407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.323369 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-071d12a3-dockercfg-rmsf7\"" Oct 30 19:56:51.324907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.324857 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd"] Oct 30 19:56:51.332113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.332092 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.332226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.332154 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.332226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.332185 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbq2\" (UniqueName: \"kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.332226 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.332214 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.332937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.332913 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.334723 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.334702 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.335291 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.335267 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.342339 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.342315 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbq2\" (UniqueName: \"kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2\") pod \"maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.363354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.363320 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl"] Oct 30 19:56:51.367155 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.367135 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.377778 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.377748 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl"] Oct 30 19:56:51.399316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.399291 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.433393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.433280 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz6rq\" (UniqueName: \"kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq\") pod \"payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.433393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.433353 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.433393 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.433385 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.433759 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.433410 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248lq\" (UniqueName: \"kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq\") pod \"payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.504083 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.504041 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:51.505014 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.504987 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.535106 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.535076 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.535237 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.535113 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248lq\" (UniqueName: \"kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq\") pod \"payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.535310 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.535293 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6rq\" (UniqueName: \"kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq\") pod \"payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.535352 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.535339 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.535850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.535827 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.536079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.536053 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.546110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.546055 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248lq\" (UniqueName: \"kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq\") pod \"payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.546326 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.546302 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6rq\" (UniqueName: \"kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq\") pod \"payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.630422 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.630387 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:51.683802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.683119 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:51.816873 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.816834 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8"] Oct 30 19:56:51.818812 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:56:51.818783 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2e3261_3211_423f_b23a_08f0a37d3f45.slice/crio-6946255907dac322f04462b658fc55a4907d9098fc3d2d5208394c9079316aa8 WatchSource:0}: Error finding container 6946255907dac322f04462b658fc55a4907d9098fc3d2d5208394c9079316aa8: Status 404 returned error can't find the container with id 6946255907dac322f04462b658fc55a4907d9098fc3d2d5208394c9079316aa8 Oct 30 19:56:51.820341 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.820325 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:56:51.912368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.912324 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd"] Oct 30 19:56:51.916797 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:56:51.916756 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3debce_f551_40b5_b738_3a5612c51eda.slice/crio-c58a5cb3fb3802184668cab8a4fa85be5424a0f366c2c02faca78f80b16bc5e3 WatchSource:0}: Error finding container c58a5cb3fb3802184668cab8a4fa85be5424a0f366c2c02faca78f80b16bc5e3: Status 404 returned error can't find the container with id c58a5cb3fb3802184668cab8a4fa85be5424a0f366c2c02faca78f80b16bc5e3 Oct 30 19:56:51.972185 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:51.972161 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl"] Oct 30 19:56:51.973744 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:56:51.973716 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd48dfb_ddc1_4681_97fe_3075eb81ab6e.slice/crio-1479482f462186bb45a4bff11ad73c3237493ea22245225c0d4579b3095822b0 WatchSource:0}: Error finding container 1479482f462186bb45a4bff11ad73c3237493ea22245225c0d4579b3095822b0: Status 404 returned error can't find the container with id 1479482f462186bb45a4bff11ad73c3237493ea22245225c0d4579b3095822b0 Oct 30 19:56:52.257966 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.257937 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:52.569308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.569277 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:56:52.616380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.614937 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:56:52.630496 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.630470 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:56:52.634491 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.634472 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:56:52.647633 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647608 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.647770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647705 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.647770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647747 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.647904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647780 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.647904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647815 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.647904 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647849 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.648081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647920 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.648081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.647972 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.648081 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.648021 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch\") pod \"99e13b5d-c7da-408f-9557-866bc278760e\" (UID: \"99e13b5d-c7da-408f-9557-866bc278760e\") " Oct 30 19:56:52.650079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.649545 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.650079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.649647 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.650079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.649789 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.650079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.649992 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:52.650079 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.650054 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.651228 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.651190 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.651607 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.651588 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "kube-api-access-xw6ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.655378 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.655339 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:52.655909 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.655826 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token" pod "99e13b5d-c7da-408f-9557-866bc278760e" (UID: "99e13b5d-c7da-408f-9557-866bc278760e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.698820 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.698782 6514 generic.go:356] "Generic (PLEG): container finished" podID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerID="d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd" exitCode=0 Oct 30 19:56:52.699046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.698837 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" Oct 30 19:56:52.699046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.698898 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" event={"ID":"c0e10d4d-bb57-4275-8929-26a9d24fe937","Type":"ContainerDied","Data":"d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd"} Oct 30 19:56:52.699046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.698953 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf" event={"ID":"c0e10d4d-bb57-4275-8929-26a9d24fe937","Type":"ContainerDied","Data":"d12453c3801c518221ff1462e13e95bd354f04ecdd9494d6be9b8b4c652a3294"} Oct 30 19:56:52.699046 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.698975 6514 scope.go:122] "RemoveContainer" containerID="d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd" Oct 30 19:56:52.701855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.701269 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" event={"ID":"8b3debce-f551-40b5-b738-3a5612c51eda","Type":"ContainerStarted","Data":"fb93e5138a3c8f28a4c6c63266eb879afd7bf72dae66cc234db7e8bdacc7e6d8"} Oct 30 19:56:52.701855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.701320 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" event={"ID":"8b3debce-f551-40b5-b738-3a5612c51eda","Type":"ContainerStarted","Data":"c58a5cb3fb3802184668cab8a4fa85be5424a0f366c2c02faca78f80b16bc5e3"} Oct 30 19:56:52.701855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.701440 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.702975 6514 generic.go:356] "Generic (PLEG): container finished" podID="99e13b5d-c7da-408f-9557-866bc278760e" containerID="0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529" exitCode=0 Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.703009 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.703063 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" event={"ID":"99e13b5d-c7da-408f-9557-866bc278760e","Type":"ContainerDied","Data":"0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529"} Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.703101 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn" event={"ID":"99e13b5d-c7da-408f-9557-866bc278760e","Type":"ContainerDied","Data":"1c15aa84169ff8d0304d544975d773dea62fd53825ce22adf8a439fc8ff85c26"} Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.704477 6514 generic.go:356] "Generic (PLEG): container finished" podID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerID="0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a" exitCode=0 Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.704607 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" event={"ID":"df818a7e-9582-43c4-ac90-2cf03bb9ddbd","Type":"ContainerDied","Data":"0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a"} Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.704644 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" event={"ID":"df818a7e-9582-43c4-ac90-2cf03bb9ddbd","Type":"ContainerDied","Data":"abd9c582a9a7f1df462b6f289193feca9296af7c0d75545c628e8dd1943e0a47"} Oct 30 19:56:52.705049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.704695 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv" Oct 30 19:56:52.706276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.706252 6514 generic.go:356] "Generic (PLEG): container finished" podID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerID="004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52" exitCode=0 Oct 30 19:56:52.706410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.706285 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" Oct 30 19:56:52.706410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.706317 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" event={"ID":"bf846e02-c35f-4f1c-933f-a5a8a4a64060","Type":"ContainerDied","Data":"004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52"} Oct 30 19:56:52.706410 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.706367 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z" event={"ID":"bf846e02-c35f-4f1c-933f-a5a8a4a64060","Type":"ContainerDied","Data":"72664eeba82d3a180e9c35671cd16b273e0c6a9d8d2b821e5089105a1856430c"} Oct 30 19:56:52.708120 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.708100 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" event={"ID":"7a2e3261-3211-423f-b23a-08f0a37d3f45","Type":"ContainerStarted","Data":"d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924"} Oct 30 19:56:52.708242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.708134 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" event={"ID":"7a2e3261-3211-423f-b23a-08f0a37d3f45","Type":"ContainerStarted","Data":"6946255907dac322f04462b658fc55a4907d9098fc3d2d5208394c9079316aa8"} Oct 30 19:56:52.708242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.708154 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:52.710059 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.710032 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" event={"ID":"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e","Type":"ContainerStarted","Data":"b219a5d657bc2e3f626077f9afa6539bc24bd73dbb1ddeedfa9ae69b4c0b4c2d"} Oct 30 19:56:52.710169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.710079 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" event={"ID":"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e","Type":"ContainerStarted","Data":"1479482f462186bb45a4bff11ad73c3237493ea22245225c0d4579b3095822b0"} Oct 30 19:56:52.710169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.710129 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:56:52.715102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.712003 6514 scope.go:122] "RemoveContainer" containerID="d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd" Oct 30 19:56:52.715102 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:56:52.712310 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd\": container with ID starting with d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd not found: ID does not exist" containerID="d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd" Oct 30 19:56:52.715102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.712337 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd"} err="failed to get container status \"d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd\": rpc error: code = NotFound desc = could not find container \"d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd\": container with ID starting with d16e3c03d001790732e795947b98cf8b2110fa5870c775c6f7901705090d00dd not found: ID does not exist" Oct 30 19:56:52.715102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.712359 6514 scope.go:122] "RemoveContainer" containerID="0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529" Oct 30 19:56:52.723604 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.723589 6514 scope.go:122] "RemoveContainer" containerID="0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529" Oct 30 19:56:52.723912 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:56:52.723891 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529\": container with ID starting with 0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529 not found: ID does not exist" containerID="0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529" Oct 30 19:56:52.723991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.723917 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529"} err="failed to get container status \"0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529\": rpc error: code = NotFound desc = could not find container \"0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529\": container with ID starting with 0982ee18be1962113c49e3fbb5386d1e971066ee1a9ee7b0e581f7ec36735529 not found: ID does not exist" Oct 30 19:56:52.723991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.723932 6514 scope.go:122] "RemoveContainer" containerID="0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a" Oct 30 19:56:52.731520 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.729927 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" podStartSLOduration=1.729912327 podStartE2EDuration="1.729912327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:52.729193657 +0000 UTC m=+2382.687012849" watchObservedRunningTime="2026-10-30 19:56:52.729912327 +0000 UTC m=+2382.687731516" Oct 30 19:56:52.733233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.733210 6514 scope.go:122] "RemoveContainer" containerID="0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a" Oct 30 19:56:52.733496 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:56:52.733479 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a\": container with ID starting with 0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a not found: ID does not exist" containerID="0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a" Oct 30 19:56:52.733547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.733504 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a"} err="failed to get container status \"0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a\": rpc error: code = NotFound desc = could not find container \"0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a\": container with ID starting with 0662a9da6cb81c6dab6ece653b7e7bf8306bda213b562a51bda23ab28a3cdb5a not found: ID does not exist" Oct 30 19:56:52.733547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.733520 6514 scope.go:122] "RemoveContainer" containerID="004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52" Oct 30 19:56:52.742752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.742724 6514 scope.go:122] "RemoveContainer" containerID="004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52" Oct 30 19:56:52.743179 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:56:52.743155 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52\": container with ID starting with 004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52 not found: ID does not exist" containerID="004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52" Oct 30 19:56:52.743276 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.743188 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52"} err="failed to get container status \"004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52\": rpc error: code = NotFound desc = could not find container \"004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52\": container with ID starting with 004b041392019fb39748e9358e2284ec851ba93f56b8859e3967f5d984d2fc52 not found: ID does not exist" Oct 30 19:56:52.749227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749207 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749240 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.749316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749259 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749316 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749283 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749409 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.749481 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749459 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.749584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749496 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.749584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749482 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.749584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749528 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749560 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.749584 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749546 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749605 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749640 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749670 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749698 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749727 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749762 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749764 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749796 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749807 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.749878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749878 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749916 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.749966 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750007 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750066 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750093 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750127 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750158 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750166 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750214 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750237 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket\") pod \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\" (UID: \"bf846e02-c35f-4f1c-933f-a5a8a4a64060\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750272 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy\") pod \"c0e10d4d-bb57-4275-8929-26a9d24fe937\" (UID: \"c0e10d4d-bb57-4275-8929-26a9d24fe937\") " Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750349 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.750379 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750376 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert\") pod \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\" (UID: \"df818a7e-9582-43c4-ac90-2cf03bb9ddbd\") " Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750506 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750701 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750911 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.750982 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751004 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6ch\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-kube-api-access-xw6ch\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751020 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751033 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751055 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751069 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751082 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751097 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751111 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751125 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751140 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.751150 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751154 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e13b5d-c7da-408f-9557-866bc278760e-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751171 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751187 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e13b5d-c7da-408f-9557-866bc278760e-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751201 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e13b5d-c7da-408f-9557-866bc278760e-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751219 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e10d4d-bb57-4275-8929-26a9d24fe937-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751234 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.751248 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e13b5d-c7da-408f-9557-866bc278760e-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752032 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752040 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752081 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.752166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752141 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:52.752847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752277 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.752847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752336 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:52.752847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752605 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:52.753052 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.752994 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "kube-api-access-xkdxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.753110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.753055 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.753299 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.753281 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "kube-api-access-hkbkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.754161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.754135 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.754302 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.754285 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.754518 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.754489 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy" pod "c0e10d4d-bb57-4275-8929-26a9d24fe937" (UID: "c0e10d4d-bb57-4275-8929-26a9d24fe937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.754645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.754627 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "kube-api-access-crbft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.754981 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.754962 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo" pod "df818a7e-9582-43c4-ac90-2cf03bb9ddbd" (UID: "df818a7e-9582-43c4-ac90-2cf03bb9ddbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:52.755169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.755152 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.755674 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.755657 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token" pod "bf846e02-c35f-4f1c-933f-a5a8a4a64060" (UID: "bf846e02-c35f-4f1c-933f-a5a8a4a64060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.763643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.763604 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" podStartSLOduration=1.763593004 podStartE2EDuration="1.763593004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:52.763003784 +0000 UTC m=+2382.720822980" watchObservedRunningTime="2026-10-30 19:56:52.763593004 +0000 UTC m=+2382.721412204" Oct 30 19:56:52.783114 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.783018 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" podStartSLOduration=1.37140915 podStartE2EDuration="1.78300598s" totalImagesPullingTime="411.59683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:52.781281323 +0000 UTC m=+2382.739100513" watchObservedRunningTime="2026-10-30 19:56:52.78300598 +0000 UTC m=+2382.740825169" Oct 30 19:56:52.800074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.800041 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn"] Oct 30 19:56:52.804107 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.804081 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s6ldn"] Oct 30 19:56:52.852424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852379 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852413 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852424 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852426 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852441 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852454 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkdxs\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-kube-api-access-xkdxs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852468 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852481 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852493 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e10d4d-bb57-4275-8929-26a9d24fe937-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852505 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852517 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbft\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-kube-api-access-crbft\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852531 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbkl\" (UniqueName: \"kubernetes.io/projected/c0e10d4d-bb57-4275-8929-26a9d24fe937-kube-api-access-hkbkl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852544 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852556 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852569 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf846e02-c35f-4f1c-933f-a5a8a4a64060-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852581 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852595 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf846e02-c35f-4f1c-933f-a5a8a4a64060-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852606 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df818a7e-9582-43c4-ac90-2cf03bb9ddbd-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.852775 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:52.852618 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e10d4d-bb57-4275-8929-26a9d24fe937-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:53.025614 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.025582 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf"] Oct 30 19:56:53.030236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.030210 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-lnkcf"] Oct 30 19:56:53.041455 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.041388 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv"] Oct 30 19:56:53.047540 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.047518 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-742kv"] Oct 30 19:56:53.054700 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.054676 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.060157 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.060136 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z"] Oct 30 19:56:53.064225 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.064201 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5jm6z"] Oct 30 19:56:53.461283 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.461252 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:56:53.757332 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:53.757297 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.157521 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:54.157445 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e13b5d-c7da-408f-9557-866bc278760e" path="/var/lib/kubelet/pods/99e13b5d-c7da-408f-9557-866bc278760e/volumes" Oct 30 19:56:54.157942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:54.157928 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" path="/var/lib/kubelet/pods/bf846e02-c35f-4f1c-933f-a5a8a4a64060/volumes" Oct 30 19:56:54.158324 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:54.158313 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" path="/var/lib/kubelet/pods/c0e10d4d-bb57-4275-8929-26a9d24fe937/volumes" Oct 30 19:56:54.159047 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:54.159027 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" path="/var/lib/kubelet/pods/df818a7e-9582-43c4-ac90-2cf03bb9ddbd/volumes" Oct 30 19:56:54.442853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:54.442803 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.191171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:55.191089 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.968575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:55.968540 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.669500 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:56.669468 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:57.462850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:57.462817 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:58.265645 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:58.265609 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:58.724746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:58.724717 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:56:59.055442 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:56:59.055359 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.137602 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:00.137565 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.854217 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:00.854181 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.750743 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:01.750706 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.562992 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:02.562953 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.249479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.249446 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.721746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.721723 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:57:03.721979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.721783 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:57:03.780603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.780519 6514 generic.go:356] "Generic (PLEG): container finished" podID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerID="a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e" exitCode=0 Oct 30 19:57:03.780603 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.780588 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" event={"ID":"dde773b7-2d55-436d-97f5-dbe701edf49a","Type":"ContainerDied","Data":"a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e"} Oct 30 19:57:03.781338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.781315 6514 scope.go:122] "RemoveContainer" containerID="a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e" Oct 30 19:57:03.782794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.782639 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:03.782794 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.782734 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:03.783031 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.782806 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:03.958122 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:03.958078 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.678578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:04.678542 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:57:04.755243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:04.755213 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.787654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:04.787623 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" event={"ID":"dde773b7-2d55-436d-97f5-dbe701edf49a","Type":"ContainerStarted","Data":"af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048"} Oct 30 19:57:04.969655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:04.969579 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.553613 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:05.553580 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.666753 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:05.666720 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.781419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:05.781389 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:57:05.782694 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:05.782665 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:57:05.782809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:05.782723 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:57:05.814388 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:05.814305 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:06.267220 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267185 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4"] Oct 30 19:57:06.267594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267578 6514 cpu_manager.go:414] "containerMap: removing container" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" Oct 30 19:57:06.267594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267591 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267612 6514 cpu_manager.go:414] "containerMap: removing container" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267618 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267658 6514 cpu_manager.go:414] "containerMap: removing container" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267663 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267673 6514 cpu_manager.go:414] "containerMap: removing container" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267678 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267737 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e13b5d-c7da-408f-9557-866bc278760e" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267747 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e10d4d-bb57-4275-8929-26a9d24fe937" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267755 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="df818a7e-9582-43c4-ac90-2cf03bb9ddbd" containerName="istio-proxy" Oct 30 19:57:06.267784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.267764 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf846e02-c35f-4f1c-933f-a5a8a4a64060" containerName="istio-proxy" Oct 30 19:57:06.271033 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.271009 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.277940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.277914 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-98b568\"/\"default-dockercfg-59zng\"" Oct 30 19:57:06.277940 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.277914 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-98b568\"/\"simulator-single-main-98b568-kserve-self-signed-certs\"" Oct 30 19:57:06.283557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.283529 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4"] Oct 30 19:57:06.334624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.334569 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:06.356360 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.356326 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-98b568\"/\"openshift-service-ca.crt\"" Oct 30 19:57:06.356526 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.356372 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-98b568\"/\"kube-root-ca.crt\"" Oct 30 19:57:06.424617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.424577 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.424617 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.424618 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.424850 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.424700 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.424937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.424851 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2swh\" (UniqueName: \"kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.424937 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.424906 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.425036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.424980 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.525784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.525701 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.525784 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.525770 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.525791 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.525823 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526068 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.525910 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2swh\" (UniqueName: \"kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526243 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.526065 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526303 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.526252 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526367 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.526294 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.526444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.526390 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.528787 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.528756 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.529018 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.529000 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.538295 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.538269 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2swh\" (UniqueName: \"kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh\") pod \"simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.557942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.557918 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:06.586910 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.586884 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:06.780919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.780826 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:57:06.780919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.780894 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:57:06.885253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:06.885228 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4"] Oct 30 19:57:06.888214 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:57:06.888184 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5cf452a_39dc_4f54_b010_67f5ec246a40.slice/crio-e119b7f79321f3aceefd4795c69bc1d04cd6f56a883c6dc6e29f03a23aa764a9 WatchSource:0}: Error finding container e119b7f79321f3aceefd4795c69bc1d04cd6f56a883c6dc6e29f03a23aa764a9: Status 404 returned error can't find the container with id e119b7f79321f3aceefd4795c69bc1d04cd6f56a883c6dc6e29f03a23aa764a9 Oct 30 19:57:07.382371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:07.382335 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.781049 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:07.781010 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:57:07.781301 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:07.781069 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:57:07.828590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:07.828552 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" event={"ID":"a5cf452a-39dc-4f54-b010-67f5ec246a40","Type":"ContainerStarted","Data":"1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae"} Oct 30 19:57:07.828781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:07.828601 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" event={"ID":"a5cf452a-39dc-4f54-b010-67f5ec246a40","Type":"ContainerStarted","Data":"e119b7f79321f3aceefd4795c69bc1d04cd6f56a883c6dc6e29f03a23aa764a9"} Oct 30 19:57:07.850197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:07.850145 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" podStartSLOduration=1.850131929 podStartE2EDuration="1.850131929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:07.849068773 +0000 UTC m=+2397.806887963" watchObservedRunningTime="2026-10-30 19:57:07.850131929 +0000 UTC m=+2397.807951117" Oct 30 19:57:08.026946 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:08.026907 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:57:08.265547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:08.265517 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.587936 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:08.587832 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:08.781074 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:08.781037 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:57:08.781260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:08.781104 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:57:09.154598 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:09.154568 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.781547 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:09.781503 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:57:09.781737 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:09.781566 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:57:09.948253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:09.948214 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.650679 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:10.650645 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.776695 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:10.776667 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:57:10.782819 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:10.782796 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 19:57:10.785377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:10.785355 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:57:10.785680 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:10.785663 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:57:10.786406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:10.786386 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:57:11.338766 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:11.338736 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.044738 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:12.044704 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.747419 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:12.747383 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.179892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:13.179851 6514 scope.go:122] "RemoveContainer" containerID="56769e3f6f913391ba7c8fb766c5cf467b917d4f4ed2475d69a4334f7112a9a5" Oct 30 19:57:13.189180 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:13.189161 6514 scope.go:122] "RemoveContainer" containerID="852dde386b3d81493500d288d2c4d1e244d94f05b7b47efcede934d1df1f7f8c" Oct 30 19:57:13.197011 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:13.196978 6514 scope.go:122] "RemoveContainer" containerID="ccc3879ba22bcf5cc6d77f907a5caa8273268b6e7fa93ab31725ef7e8307e5ef" Oct 30 19:57:13.275421 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:13.275387 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.449624 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:13.449547 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:14.153176 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:14.153142 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:14.852127 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:14.852098 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.704389 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:15.704351 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:16.437328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:16.437291 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:16.587028 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:16.586997 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:16.599934 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:16.599910 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:16.883851 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:16.883772 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:57:17.153652 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.153542 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.384636 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.384600 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6"] Oct 30 19:57:17.389829 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.389806 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.399736 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.399712 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gt4zp\"" Oct 30 19:57:17.408364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.408304 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6"] Oct 30 19:57:17.429273 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429243 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1db0f-3030-450b-8a45-83b100e58823-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429404 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429296 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429470 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429444 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429544 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429481 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1db0f-3030-450b-8a45-83b100e58823-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429576 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1db0f-3030-450b-8a45-83b100e58823-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429630 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429726 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429703 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429748 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhpj2\" (UniqueName: \"kubernetes.io/projected/91f1db0f-3030-450b-8a45-83b100e58823-kube-api-access-rhpj2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.429824 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.429779 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.530846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.530815 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.530846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.530848 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1db0f-3030-450b-8a45-83b100e58823-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.530919 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1db0f-3030-450b-8a45-83b100e58823-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.530949 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531086 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531119 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpj2\" (UniqueName: \"kubernetes.io/projected/91f1db0f-3030-450b-8a45-83b100e58823-kube-api-access-rhpj2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531161 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531188 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1db0f-3030-450b-8a45-83b100e58823-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531222 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531354 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531399 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531380 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531463 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531517 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.531949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.531911 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f1db0f-3030-450b-8a45-83b100e58823-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.533473 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.533443 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f1db0f-3030-450b-8a45-83b100e58823-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.533618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.533453 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f1db0f-3030-450b-8a45-83b100e58823-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.540098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.540073 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpj2\" (UniqueName: \"kubernetes.io/projected/91f1db0f-3030-450b-8a45-83b100e58823-kube-api-access-rhpj2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.540269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.540251 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f1db0f-3030-450b-8a45-83b100e58823-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qbqj6\" (UID: \"91f1db0f-3030-450b-8a45-83b100e58823\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.702773 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.702744 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:17.952449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.952379 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.987357 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.987320 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6"] Oct 30 19:57:17.989806 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:57:17.989779 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f1db0f_3030_450b_8a45_83b100e58823.slice/crio-ec03b9d5f9382c350c0534ddab90a87b2ee3f048ebfbab86aceb8c819aa0a075 WatchSource:0}: Error finding container ec03b9d5f9382c350c0534ddab90a87b2ee3f048ebfbab86aceb8c819aa0a075: Status 404 returned error can't find the container with id ec03b9d5f9382c350c0534ddab90a87b2ee3f048ebfbab86aceb8c819aa0a075 Oct 30 19:57:17.992166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.992136 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:17.992249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.992198 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:17.992249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:17.992225 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:57:18.766098 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:18.766061 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.885882 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:18.885826 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" event={"ID":"91f1db0f-3030-450b-8a45-83b100e58823","Type":"ContainerStarted","Data":"ee4959e16b8b5f1f9b13d7eeb821198cdd7bf69ab4fbe4ebb25a9f39610ba7e3"} Oct 30 19:57:18.886151 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:18.885910 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" event={"ID":"91f1db0f-3030-450b-8a45-83b100e58823","Type":"ContainerStarted","Data":"ec03b9d5f9382c350c0534ddab90a87b2ee3f048ebfbab86aceb8c819aa0a075"} Oct 30 19:57:18.934644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:18.934577 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podStartSLOduration=1.934558892 podStartE2EDuration="1.934558892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:18.933079349 +0000 UTC m=+2408.890898539" watchObservedRunningTime="2026-10-30 19:57:18.934558892 +0000 UTC m=+2408.892378081" Oct 30 19:57:19.540249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:19.540211 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.709451 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:19.705157 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:19.712884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:19.710981 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:57:19.712884 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:19.711037 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 19:57:20.266169 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:20.266141 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.716013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:20.715968 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:57:20.716206 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:20.716030 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 19:57:20.883159 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:57:20.883120 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:57:21.150827 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:21.150732 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.709888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:21.709258 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:57:21.709888 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:21.709332 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 19:57:21.933174 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:21.933145 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.708260 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:22.708219 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:57:22.708600 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:22.708569 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 19:57:22.865605 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:22.865557 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.711886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:23.711264 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:57:23.711886 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:23.711332 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 19:57:23.764075 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:23.764018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.549716 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:24.549679 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.703576 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:24.703536 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:57:24.703746 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:24.703613 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 19:57:25.421163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:25.421124 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:25.629244 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:25.629208 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:57:26.259972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:26.259938 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.284848 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:26.284818 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:26.285364 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:26.285333 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:26.286677 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:26.286654 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" Oct 30 19:57:27.053466 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:27.053433 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.838590 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:27.838557 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:28.577358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:28.577316 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:29.244533 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:29.244502 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.043262 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:30.043223 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.800783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:30.800752 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:31.517261 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:31.517215 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.232558 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:32.232522 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.936099 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:32.936069 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.646104 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:33.646067 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.335612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:34.335576 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:35.050809 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:35.050773 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:35.740109 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:35.740058 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:36.437171 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:36.437135 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:37.156312 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:37.156239 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:37.858625 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:37.858519 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:38.715899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:38.715830 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.551542 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:39.551492 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:40.334852 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:40.334821 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.062822 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:41.062791 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.894164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:41.894132 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.550398 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:42.550364 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.268269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:43.268234 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:44.038921 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:44.038893 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:44.750245 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:44.750212 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:45.445233 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:45.445173 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.166373 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:46.166339 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.845264 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:46.845227 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.550354 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:47.550318 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.723057 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:47.723018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.461608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:48.461555 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.479471 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:48.479437 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:57:49.391643 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:49.391610 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.059646 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:50.059616 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.757197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:50.757162 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.883930 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:57:50.882840 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:57:51.560008 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:51.559979 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.245654 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:52.245571 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.665050 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:52.664965 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4"] Oct 30 19:57:52.772242 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:52.772199 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.955322 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:52.955284 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.649479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:53.649447 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:54.427210 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:54.427179 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:55.159259 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:55.159221 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:55.843064 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:55.843024 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.544513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:56.544471 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.237143 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:57.237109 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.942377 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:57.942347 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.988055 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:57.988021 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl"] Oct 30 19:57:58.003843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.003814 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd"] Oct 30 19:57:58.076144 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.076092 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.194201 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.194123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.468907 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.468773 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8"] Oct 30 19:57:58.469102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.469060 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" podUID="7a2e3261-3211-423f-b23a-08f0a37d3f45" containerName="maas-api" containerID="cri-o://d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924" gracePeriod=30 Oct 30 19:57:58.544561 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.544528 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.698976 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.698943 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.720338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.720255 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" podUID="7a2e3261-3211-423f-b23a-08f0a37d3f45" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.174:8443/health\": dial tcp 10.134.0.174:8443: connect: connection refused" Oct 30 19:57:58.817255 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.817179 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.920996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.920967 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:57:58.932319 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.932295 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2\" (UniqueName: \"kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2\") pod \"7a2e3261-3211-423f-b23a-08f0a37d3f45\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " Oct 30 19:57:58.932438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.932347 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs\") pod \"7a2e3261-3211-423f-b23a-08f0a37d3f45\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " Oct 30 19:57:58.932438 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.932413 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca\") pod \"7a2e3261-3211-423f-b23a-08f0a37d3f45\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " Oct 30 19:57:58.932513 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.932443 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls\") pod \"7a2e3261-3211-423f-b23a-08f0a37d3f45\" (UID: \"7a2e3261-3211-423f-b23a-08f0a37d3f45\") " Oct 30 19:57:58.932781 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.932750 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca" pod "7a2e3261-3211-423f-b23a-08f0a37d3f45" (UID: "7a2e3261-3211-423f-b23a-08f0a37d3f45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:58.934465 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.934436 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2" pod "7a2e3261-3211-423f-b23a-08f0a37d3f45" (UID: "7a2e3261-3211-423f-b23a-08f0a37d3f45"). InnerVolumeSpecName "kube-api-access-9hbq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:58.934939 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.934911 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls" pod "7a2e3261-3211-423f-b23a-08f0a37d3f45" (UID: "7a2e3261-3211-423f-b23a-08f0a37d3f45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:58.935029 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:58.934933 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs" pod "7a2e3261-3211-423f-b23a-08f0a37d3f45" (UID: "7a2e3261-3211-423f-b23a-08f0a37d3f45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:59.033711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.033623 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbq2\" (UniqueName: \"kubernetes.io/projected/7a2e3261-3211-423f-b23a-08f0a37d3f45-kube-api-access-9hbq2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.033711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.033654 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.033711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.033665 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2e3261-3211-423f-b23a-08f0a37d3f45-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.033711 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.033674 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2e3261-3211-423f-b23a-08f0a37d3f45-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.106803 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.106764 6514 generic.go:356] "Generic (PLEG): container finished" podID="7a2e3261-3211-423f-b23a-08f0a37d3f45" containerID="d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924" exitCode=0 Oct 30 19:57:59.107003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.106819 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" Oct 30 19:57:59.107003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.106854 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" event={"ID":"7a2e3261-3211-423f-b23a-08f0a37d3f45","Type":"ContainerDied","Data":"d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924"} Oct 30 19:57:59.107003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.106927 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8" event={"ID":"7a2e3261-3211-423f-b23a-08f0a37d3f45","Type":"ContainerDied","Data":"6946255907dac322f04462b658fc55a4907d9098fc3d2d5208394c9079316aa8"} Oct 30 19:57:59.107003 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.106950 6514 scope.go:122] "RemoveContainer" containerID="d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924" Oct 30 19:57:59.117382 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.117294 6514 scope.go:122] "RemoveContainer" containerID="d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924" Oct 30 19:57:59.117629 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:57:59.117605 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924\": container with ID starting with d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924 not found: ID does not exist" containerID="d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924" Oct 30 19:57:59.117681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.117636 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924"} err="failed to get container status \"d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924\": rpc error: code = NotFound desc = could not find container \"d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924\": container with ID starting with d13028e81cdab58279161e1d955679fece6202f74e683a73683a3ca04914e924 not found: ID does not exist" Oct 30 19:57:59.147247 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.147221 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8"] Oct 30 19:57:59.162968 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.162944 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98b568-54cc897f44-8l9p8"] Oct 30 19:57:59.331267 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:57:59.331197 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.157428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:00.157384 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2e3261-3211-423f-b23a-08f0a37d3f45" path="/var/lib/kubelet/pods/7a2e3261-3211-423f-b23a-08f0a37d3f45/volumes" Oct 30 19:58:00.174783 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:00.174753 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:58:02.988591 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:02.988554 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerName="payload-processing" containerID="cri-o://b219a5d657bc2e3f626077f9afa6539bc24bd73dbb1ddeedfa9ae69b4c0b4c2d" gracePeriod=25 Oct 30 19:58:03.004996 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:03.004956 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" containerName="payload-pre-processing" containerID="cri-o://fb93e5138a3c8f28a4c6c63266eb879afd7bf72dae66cc234db7e8bdacc7e6d8" gracePeriod=25 Oct 30 19:58:03.720644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:03.720601 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:03.720644 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:03.720622 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:05.633358 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:05.633311 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.755813 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:05.755774 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.925187 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:05.925056 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr"] Oct 30 19:58:05.925406 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:05.925257 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" containerID="cri-o://af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048" gracePeriod=30 Oct 30 19:58:06.004203 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:06.004127 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.169468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.169430 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl"] Oct 30 19:58:07.171919 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.171896 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8vwl"] Oct 30 19:58:07.182583 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.182556 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6"] Oct 30 19:58:07.182986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.182972 6514 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e3261-3211-423f-b23a-08f0a37d3f45" containerName="maas-api" Oct 30 19:58:07.183036 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.182987 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e3261-3211-423f-b23a-08f0a37d3f45" containerName="maas-api" Oct 30 19:58:07.183111 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.183101 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2e3261-3211-423f-b23a-08f0a37d3f45" containerName="maas-api" Oct 30 19:58:07.187369 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.187352 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.195132 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.194108 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:58:07.195348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.195325 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:07.195348 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.195335 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:58:07.195527 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.195415 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:58:07.197253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.197230 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6"] Oct 30 19:58:07.313993 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.313954 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.314165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.314027 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmvz\" (UniqueName: \"kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.314165 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.314119 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.415408 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.415370 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.415578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.415420 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.415578 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.415466 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmvz\" (UniqueName: \"kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.416125 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.416099 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.417681 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.417654 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.424434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.424384 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmvz\" (UniqueName: \"kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz\") pod \"maas-api-revoke-sub-3d4156f1-wh2b6\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.501229 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.501195 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:07.685198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.685163 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" podUID="a5cf452a-39dc-4f54-b010-67f5ec246a40" containerName="main" containerID="cri-o://1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae" gracePeriod=45 Oct 30 19:58:07.772776 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.772752 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6"] Oct 30 19:58:07.775482 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:58:07.775446 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540a3e30_6c89_4ca5_ad58_b870efa3c87e.slice/crio-3a3ba883c5268a053f1d1c75e6875c7837f50211b30b3c57e725e3b9a9a7a896 WatchSource:0}: Error finding container 3a3ba883c5268a053f1d1c75e6875c7837f50211b30b3c57e725e3b9a9a7a896: Status 404 returned error can't find the container with id 3a3ba883c5268a053f1d1c75e6875c7837f50211b30b3c57e725e3b9a9a7a896 Oct 30 19:58:07.920817 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:07.920789 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:58:08.020892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.020853 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir\") pod \"a5cf452a-39dc-4f54-b010-67f5ec246a40\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " Oct 30 19:58:08.021053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.020908 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm\") pod \"a5cf452a-39dc-4f54-b010-67f5ec246a40\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " Oct 30 19:58:08.021053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.020967 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh\" (UniqueName: \"kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh\") pod \"a5cf452a-39dc-4f54-b010-67f5ec246a40\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " Oct 30 19:58:08.021053 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.020999 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs\") pod \"a5cf452a-39dc-4f54-b010-67f5ec246a40\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " Oct 30 19:58:08.021282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021063 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home\") pod \"a5cf452a-39dc-4f54-b010-67f5ec246a40\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " Oct 30 19:58:08.021282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021079 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir" pod "a5cf452a-39dc-4f54-b010-67f5ec246a40" (UID: "a5cf452a-39dc-4f54-b010-67f5ec246a40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:08.021282 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021123 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache\") pod \"a5cf452a-39dc-4f54-b010-67f5ec246a40\" (UID: \"a5cf452a-39dc-4f54-b010-67f5ec246a40\") " Oct 30 19:58:08.021413 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021304 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home" pod "a5cf452a-39dc-4f54-b010-67f5ec246a40" (UID: "a5cf452a-39dc-4f54-b010-67f5ec246a40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:08.021462 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021421 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache" pod "a5cf452a-39dc-4f54-b010-67f5ec246a40" (UID: "a5cf452a-39dc-4f54-b010-67f5ec246a40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:08.021551 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021535 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:08.021589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021556 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:08.021589 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.021569 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:08.023541 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.023516 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs" pod "a5cf452a-39dc-4f54-b010-67f5ec246a40" (UID: "a5cf452a-39dc-4f54-b010-67f5ec246a40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:08.023672 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.023621 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm" pod "a5cf452a-39dc-4f54-b010-67f5ec246a40" (UID: "a5cf452a-39dc-4f54-b010-67f5ec246a40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:08.023741 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.023728 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh" pod "a5cf452a-39dc-4f54-b010-67f5ec246a40" (UID: "a5cf452a-39dc-4f54-b010-67f5ec246a40"). InnerVolumeSpecName "kube-api-access-c2swh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:08.122254 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.122232 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2swh\" (UniqueName: \"kubernetes.io/projected/a5cf452a-39dc-4f54-b010-67f5ec246a40-kube-api-access-c2swh\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:08.122363 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.122256 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cf452a-39dc-4f54-b010-67f5ec246a40-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:08.122363 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.122266 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cf452a-39dc-4f54-b010-67f5ec246a40-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:08.154484 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.154452 6514 generic.go:356] "Generic (PLEG): container finished" podID="a5cf452a-39dc-4f54-b010-67f5ec246a40" containerID="1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae" exitCode=0 Oct 30 19:58:08.154621 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.154556 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" Oct 30 19:58:08.157785 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.157762 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c054dead-d221-4741-8ea4-d79d1204158f" path="/var/lib/kubelet/pods/c054dead-d221-4741-8ea4-d79d1204158f/volumes" Oct 30 19:58:08.158181 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.158163 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" event={"ID":"540a3e30-6c89-4ca5-ad58-b870efa3c87e","Type":"ContainerStarted","Data":"5546578811ee110cbc79e3db934a161ea47360297956f44d5023ecdcfae22f0e"} Oct 30 19:58:08.158272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.158196 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" event={"ID":"540a3e30-6c89-4ca5-ad58-b870efa3c87e","Type":"ContainerStarted","Data":"3a3ba883c5268a053f1d1c75e6875c7837f50211b30b3c57e725e3b9a9a7a896"} Oct 30 19:58:08.158272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.158213 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" event={"ID":"a5cf452a-39dc-4f54-b010-67f5ec246a40","Type":"ContainerDied","Data":"1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae"} Oct 30 19:58:08.158272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.158226 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" event={"ID":"a5cf452a-39dc-4f54-b010-67f5ec246a40","Type":"ContainerDied","Data":"e119b7f79321f3aceefd4795c69bc1d04cd6f56a883c6dc6e29f03a23aa764a9"} Oct 30 19:58:08.158272 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.158241 6514 scope.go:122] "RemoveContainer" containerID="1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae" Oct 30 19:58:08.167013 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.166958 6514 scope.go:122] "RemoveContainer" containerID="1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae" Oct 30 19:58:08.167238 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:58:08.167220 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae\": container with ID starting with 1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae not found: ID does not exist" containerID="1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae" Oct 30 19:58:08.167305 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.167243 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae"} err="failed to get container status \"1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae\": rpc error: code = NotFound desc = could not find container \"1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae\": container with ID starting with 1ca3f9e839c98174bb10b29675b0d2aba1addf958c8d0f2897b10f4487665dae not found: ID does not exist" Oct 30 19:58:08.171240 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.171202 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" podStartSLOduration=1.17119218 podStartE2EDuration="1.17119218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:08.170079978 +0000 UTC m=+2458.127899167" watchObservedRunningTime="2026-10-30 19:58:08.17119218 +0000 UTC m=+2458.129011369" Oct 30 19:58:08.182986 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.182965 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4"] Oct 30 19:58:08.193855 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.193832 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4"] Oct 30 19:58:08.873760 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:08.873724 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-98b568/simulator-single-main-98b568-kserve-7dcd5ccb44-wbhf4" podUID="a5cf452a-39dc-4f54-b010-67f5ec246a40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.177:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:09.162898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:09.161376 6514 generic.go:356] "Generic (PLEG): container finished" podID="540a3e30-6c89-4ca5-ad58-b870efa3c87e" containerID="5546578811ee110cbc79e3db934a161ea47360297956f44d5023ecdcfae22f0e" exitCode=0 Oct 30 19:58:09.162898 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:09.161483 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" event={"ID":"540a3e30-6c89-4ca5-ad58-b870efa3c87e","Type":"ContainerDied","Data":"5546578811ee110cbc79e3db934a161ea47360297956f44d5023ecdcfae22f0e"} Oct 30 19:58:10.157594 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.157561 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cf452a-39dc-4f54-b010-67f5ec246a40" path="/var/lib/kubelet/pods/a5cf452a-39dc-4f54-b010-67f5ec246a40/volumes" Oct 30 19:58:10.301769 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.301746 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:10.441673 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.441647 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca\") pod \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " Oct 30 19:58:10.441853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.441711 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz\" (UniqueName: \"kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz\") pod \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " Oct 30 19:58:10.441853 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.441832 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca\") pod \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\" (UID: \"540a3e30-6c89-4ca5-ad58-b870efa3c87e\") " Oct 30 19:58:10.442210 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.442175 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca" pod "540a3e30-6c89-4ca5-ad58-b870efa3c87e" (UID: "540a3e30-6c89-4ca5-ad58-b870efa3c87e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:10.443847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.443825 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca" pod "540a3e30-6c89-4ca5-ad58-b870efa3c87e" (UID: "540a3e30-6c89-4ca5-ad58-b870efa3c87e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:10.443969 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.443846 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz" pod "540a3e30-6c89-4ca5-ad58-b870efa3c87e" (UID: "540a3e30-6c89-4ca5-ad58-b870efa3c87e"). InnerVolumeSpecName "kube-api-access-njmvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:10.543126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.543095 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:10.543126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.543122 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a3e30-6c89-4ca5-ad58-b870efa3c87e-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:10.543126 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.543131 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njmvz\" (UniqueName: \"kubernetes.io/projected/540a3e30-6c89-4ca5-ad58-b870efa3c87e-kube-api-access-njmvz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:10.788837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.788759 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:58:10.788837 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:10.788810 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:58:11.158110 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.158085 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:58:11.173806 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.173776 6514 generic.go:356] "Generic (PLEG): container finished" podID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerID="af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048" exitCode=0 Oct 30 19:58:11.173952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.173822 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" Oct 30 19:58:11.173952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.173886 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" event={"ID":"dde773b7-2d55-436d-97f5-dbe701edf49a","Type":"ContainerDied","Data":"af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048"} Oct 30 19:58:11.173952 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.173940 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr" event={"ID":"dde773b7-2d55-436d-97f5-dbe701edf49a","Type":"ContainerDied","Data":"cd1b826a68e69b65b3dec460ba65e6c520c1db38c0f0e121b6cff1fcb2d673c9"} Oct 30 19:58:11.174130 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.173956 6514 scope.go:122] "RemoveContainer" containerID="af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048" Oct 30 19:58:11.175131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.175012 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" event={"ID":"540a3e30-6c89-4ca5-ad58-b870efa3c87e","Type":"ContainerDied","Data":"3a3ba883c5268a053f1d1c75e6875c7837f50211b30b3c57e725e3b9a9a7a896"} Oct 30 19:58:11.175131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.175056 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a3ba883c5268a053f1d1c75e6875c7837f50211b30b3c57e725e3b9a9a7a896" Oct 30 19:58:11.175131 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.175063 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6" Oct 30 19:58:11.184468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.184450 6514 scope.go:122] "RemoveContainer" containerID="a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e" Oct 30 19:58:11.194508 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.194493 6514 scope.go:122] "RemoveContainer" containerID="af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048" Oct 30 19:58:11.194806 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:58:11.194782 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048\": container with ID starting with af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048 not found: ID does not exist" containerID="af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048" Oct 30 19:58:11.194899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.194817 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048"} err="failed to get container status \"af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048\": rpc error: code = NotFound desc = could not find container \"af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048\": container with ID starting with af45dfd13dd3b5a46417f27c461dd5ec17e58a079b60f7e24e68d6e745fac048 not found: ID does not exist" Oct 30 19:58:11.194899 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.194840 6514 scope.go:122] "RemoveContainer" containerID="a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e" Oct 30 19:58:11.195156 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:58:11.195137 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e\": container with ID starting with a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e not found: ID does not exist" containerID="a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e" Oct 30 19:58:11.195198 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.195162 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e"} err="failed to get container status \"a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e\": rpc error: code = NotFound desc = could not find container \"a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e\": container with ID starting with a8c398d50e6a76f4a5c3509a593128175e8e8a56861cfdb073fc6916c698b29e not found: ID does not exist" Oct 30 19:58:11.250306 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250285 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250319 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250338 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250385 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250428 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250411 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250440 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250561 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250623 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.250690 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250640 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:11.251241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250656 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket\") pod \"dde773b7-2d55-436d-97f5-dbe701edf49a\" (UID: \"dde773b7-2d55-436d-97f5-dbe701edf49a\") " Oct 30 19:58:11.251241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250821 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:11.251241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.250845 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:11.251241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.251157 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:11.251241 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.251210 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:11.251479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.251270 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.251479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.251293 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.251479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.251307 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde773b7-2d55-436d-97f5-dbe701edf49a-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.251479 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.251320 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.252847 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.252828 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:11.253102 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.253074 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:11.253195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.253080 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:11.253195 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.253116 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp" pod "dde773b7-2d55-436d-97f5-dbe701edf49a" (UID: "dde773b7-2d55-436d-97f5-dbe701edf49a"). InnerVolumeSpecName "kube-api-access-shfwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:11.352010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.351947 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfwp\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-kube-api-access-shfwp\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.352010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.351970 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.352010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.351981 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.352010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.351989 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.352010 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.351998 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde773b7-2d55-436d-97f5-dbe701edf49a-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:11.503752 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.503723 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr"] Oct 30 19:58:11.507903 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:11.507878 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98b568-openshift-default-96865bb7c-bxslr"] Oct 30 19:58:12.158197 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:12.158169 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" path="/var/lib/kubelet/pods/dde773b7-2d55-436d-97f5-dbe701edf49a/volumes" Oct 30 19:58:13.319557 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:13.319499 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.720474 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:13.720438 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:13.720635 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:13.720440 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:18.126846 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:18.126807 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:58:20.882116 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:58:20.882076 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:58:21.977030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:21.976997 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:58:23.235142 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.235101 6514 generic.go:356] "Generic (PLEG): container finished" podID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerID="b219a5d657bc2e3f626077f9afa6539bc24bd73dbb1ddeedfa9ae69b4c0b4c2d" exitCode=0 Oct 30 19:58:23.235550 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.235165 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" event={"ID":"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e","Type":"ContainerDied","Data":"b219a5d657bc2e3f626077f9afa6539bc24bd73dbb1ddeedfa9ae69b4c0b4c2d"} Oct 30 19:58:23.237038 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.237015 6514 generic.go:356] "Generic (PLEG): container finished" podID="8b3debce-f551-40b5-b738-3a5612c51eda" containerID="fb93e5138a3c8f28a4c6c63266eb879afd7bf72dae66cc234db7e8bdacc7e6d8" exitCode=0 Oct 30 19:58:23.237163 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.237045 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" event={"ID":"8b3debce-f551-40b5-b738-3a5612c51eda","Type":"ContainerDied","Data":"fb93e5138a3c8f28a4c6c63266eb879afd7bf72dae66cc234db7e8bdacc7e6d8"} Oct 30 19:58:23.256991 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.256971 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:58:23.260038 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.260022 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:58:23.359042 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359004 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume\") pod \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " Oct 30 19:58:23.359253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359059 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume\") pod \"8b3debce-f551-40b5-b738-3a5612c51eda\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " Oct 30 19:58:23.359253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359097 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq\" (UniqueName: \"kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq\") pod \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\" (UID: \"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e\") " Oct 30 19:58:23.359253 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359179 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq\" (UniqueName: \"kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq\") pod \"8b3debce-f551-40b5-b738-3a5612c51eda\" (UID: \"8b3debce-f551-40b5-b738-3a5612c51eda\") " Oct 30 19:58:23.359436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359382 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume" pod "7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" (UID: "7fd48dfb-ddc1-4681-97fe-3075eb81ab6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:23.359499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359482 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:23.359553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.359490 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume" pod "8b3debce-f551-40b5-b738-3a5612c51eda" (UID: "8b3debce-f551-40b5-b738-3a5612c51eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:23.361278 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.361253 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq" pod "8b3debce-f551-40b5-b738-3a5612c51eda" (UID: "8b3debce-f551-40b5-b738-3a5612c51eda"). InnerVolumeSpecName "kube-api-access-jz6rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:23.361371 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.361303 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq" pod "7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" (UID: "7fd48dfb-ddc1-4681-97fe-3075eb81ab6e"). InnerVolumeSpecName "kube-api-access-248lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:23.460543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.460511 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b3debce-f551-40b5-b738-3a5612c51eda-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:23.460543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.460539 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248lq\" (UniqueName: \"kubernetes.io/projected/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e-kube-api-access-248lq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:23.460543 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:23.460549 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz6rq\" (UniqueName: \"kubernetes.io/projected/8b3debce-f551-40b5-b738-3a5612c51eda-kube-api-access-jz6rq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:24.242906 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.242851 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" Oct 30 19:58:24.243328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.242894 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl" event={"ID":"7fd48dfb-ddc1-4681-97fe-3075eb81ab6e","Type":"ContainerDied","Data":"1479482f462186bb45a4bff11ad73c3237493ea22245225c0d4579b3095822b0"} Oct 30 19:58:24.243328 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.242974 6514 scope.go:122] "RemoveContainer" containerID="b219a5d657bc2e3f626077f9afa6539bc24bd73dbb1ddeedfa9ae69b4c0b4c2d" Oct 30 19:58:24.244449 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.244431 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" Oct 30 19:58:24.244517 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.244459 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd" event={"ID":"8b3debce-f551-40b5-b738-3a5612c51eda","Type":"ContainerDied","Data":"c58a5cb3fb3802184668cab8a4fa85be5424a0f366c2c02faca78f80b16bc5e3"} Oct 30 19:58:24.252353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.252337 6514 scope.go:122] "RemoveContainer" containerID="fb93e5138a3c8f28a4c6c63266eb879afd7bf72dae66cc234db7e8bdacc7e6d8" Oct 30 19:58:24.261113 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.261094 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd"] Oct 30 19:58:24.265164 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.265140 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-98b568-58446m98kd"] Oct 30 19:58:24.276597 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.276571 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl"] Oct 30 19:58:24.280314 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:24.280294 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-98b568-75c89f78cgkxbl"] Oct 30 19:58:26.157249 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:26.157218 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" path="/var/lib/kubelet/pods/7fd48dfb-ddc1-4681-97fe-3075eb81ab6e/volumes" Oct 30 19:58:26.157916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:26.157584 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" path="/var/lib/kubelet/pods/8b3debce-f551-40b5-b738-3a5612c51eda/volumes" Oct 30 19:58:29.596664 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:29.596628 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.394436 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:30.394388 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.130115 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:36.130079 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.527612 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.527573 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf"] Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528003 6514 cpu_manager.go:414] "containerMap: removing container" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528018 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528033 6514 cpu_manager.go:414] "containerMap: removing container" podUID="a5cf452a-39dc-4f54-b010-67f5ec246a40" containerName="main" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528041 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cf452a-39dc-4f54-b010-67f5ec246a40" containerName="main" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528049 6514 cpu_manager.go:414] "containerMap: removing container" podUID="540a3e30-6c89-4ca5-ad58-b870efa3c87e" containerName="revoke-keys" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528053 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540a3e30-6c89-4ca5-ad58-b870efa3c87e" containerName="revoke-keys" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528076 6514 cpu_manager.go:414] "containerMap: removing container" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" containerName="payload-pre-processing" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528081 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" containerName="payload-pre-processing" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528094 6514 cpu_manager.go:414] "containerMap: removing container" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerName="payload-processing" Oct 30 19:58:47.528096 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528099 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerName="payload-processing" Oct 30 19:58:47.528593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528162 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" Oct 30 19:58:47.528593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528171 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cf452a-39dc-4f54-b010-67f5ec246a40" containerName="main" Oct 30 19:58:47.528593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528180 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="540a3e30-6c89-4ca5-ad58-b870efa3c87e" containerName="revoke-keys" Oct 30 19:58:47.528593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528194 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3debce-f551-40b5-b738-3a5612c51eda" containerName="payload-pre-processing" Oct 30 19:58:47.528593 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.528203 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd48dfb-ddc1-4681-97fe-3075eb81ab6e" containerName="payload-processing" Oct 30 19:58:47.531420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.531402 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.550802 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.550768 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf"] Oct 30 19:58:47.559979 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.559946 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560031 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560121 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560090 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560213 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560161 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560263 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560228 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560309 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560270 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88856f6f-b914-4487-9da6-5107913dc38e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560383 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560363 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88856f6f-b914-4487-9da6-5107913dc38e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560439 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560425 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5lt\" (UniqueName: \"kubernetes.io/projected/88856f6f-b914-4487-9da6-5107913dc38e-kube-api-access-tb5lt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.560489 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.560470 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88856f6f-b914-4487-9da6-5107913dc38e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.660948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.660908 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.660948 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.660952 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.660990 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661009 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661026 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88856f6f-b914-4487-9da6-5107913dc38e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661073 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88856f6f-b914-4487-9da6-5107913dc38e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661132 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5lt\" (UniqueName: \"kubernetes.io/projected/88856f6f-b914-4487-9da6-5107913dc38e-kube-api-access-tb5lt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661172 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88856f6f-b914-4487-9da6-5107913dc38e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661227 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661214 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661437 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661480 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661526 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.661637 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.661592 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.662040 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.662018 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88856f6f-b914-4487-9da6-5107913dc38e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.663363 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.663333 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88856f6f-b914-4487-9da6-5107913dc38e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.663488 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.663468 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88856f6f-b914-4487-9da6-5107913dc38e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.671238 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.671184 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88856f6f-b914-4487-9da6-5107913dc38e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.671368 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.671340 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5lt\" (UniqueName: \"kubernetes.io/projected/88856f6f-b914-4487-9da6-5107913dc38e-kube-api-access-tb5lt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-82vrf\" (UID: \"88856f6f-b914-4487-9da6-5107913dc38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:47.854380 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:47.854299 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:48.136892 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.136852 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf"] Oct 30 19:58:48.139621 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:58:48.139595 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88856f6f_b914_4487_9da6_5107913dc38e.slice/crio-aac873cf17678872c5a70e3825c118621994533bb6a1b3850936ec71f3700d4d WatchSource:0}: Error finding container aac873cf17678872c5a70e3825c118621994533bb6a1b3850936ec71f3700d4d: Status 404 returned error can't find the container with id aac873cf17678872c5a70e3825c118621994533bb6a1b3850936ec71f3700d4d Oct 30 19:58:48.142338 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.142304 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:48.142458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.142385 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:48.142458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.142428 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:48.353066 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.353022 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" event={"ID":"88856f6f-b914-4487-9da6-5107913dc38e","Type":"ContainerStarted","Data":"f9808cebfd76b62971877c65dd55c3d751c0d1b9c3cc8f9068f631fc082cf271"} Oct 30 19:58:48.353236 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.353093 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" event={"ID":"88856f6f-b914-4487-9da6-5107913dc38e","Type":"ContainerStarted","Data":"aac873cf17678872c5a70e3825c118621994533bb6a1b3850936ec71f3700d4d"} Oct 30 19:58:48.375563 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.375515 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podStartSLOduration=1.375501549 podStartE2EDuration="1.375501549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:48.374007164 +0000 UTC m=+2498.331826353" watchObservedRunningTime="2026-10-30 19:58:48.375501549 +0000 UTC m=+2498.333320738" Oct 30 19:58:48.854468 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:48.854435 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:49.860778 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:49.860750 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:50.362446 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:50.362417 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:50.363486 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:50.363464 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 19:58:50.880831 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:58:50.880791 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:58:54.048161 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:54.048120 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.157972 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:58:54.157938 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:02.524356 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.524317 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw"] Oct 30 19:59:02.524828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.524763 6514 cpu_manager.go:414] "containerMap: removing container" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" Oct 30 19:59:02.524828 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.524777 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" Oct 30 19:59:02.524942 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.524859 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde773b7-2d55-436d-97f5-dbe701edf49a" containerName="istio-proxy" Oct 30 19:59:02.528020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.528001 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.540916 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.540886 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw"] Oct 30 19:59:02.568620 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.568588 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f"] Oct 30 19:59:02.572420 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.572397 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.584362 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.584339 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f"] Oct 30 19:59:02.599549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599516 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.599698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599568 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqfs\" (UniqueName: \"kubernetes.io/projected/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-kube-api-access-rbqfs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.599698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599646 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.599698 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599689 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.599928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599743 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.599928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599765 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.599928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599806 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.599928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599829 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.599928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599877 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69def90a-99a6-4c50-a5a0-a9182330ff80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.599928 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599916 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599936 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.599987 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.600009 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.600025 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.600041 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.600132 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.600166 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.600288 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.600195 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7ph\" (UniqueName: \"kubernetes.io/projected/69def90a-99a6-4c50-a5a0-a9182330ff80-kube-api-access-gc7ph\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.700965 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701006 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701019 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701042 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701072 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701096 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701125 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69def90a-99a6-4c50-a5a0-a9182330ff80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701250 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701211 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701258 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701304 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701338 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701372 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701399 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701455 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701460 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701483 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.701535 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701529 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7ph\" (UniqueName: \"kubernetes.io/projected/69def90a-99a6-4c50-a5a0-a9182330ff80-kube-api-access-gc7ph\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701588 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701627 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701665 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqfs\" (UniqueName: \"kubernetes.io/projected/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-kube-api-access-rbqfs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701693 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701694 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701710 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701768 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.702020 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.701898 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69def90a-99a6-4c50-a5a0-a9182330ff80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.702472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.702038 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.702072 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.702472 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.702418 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.702608 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.702508 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.703967 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.703931 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.704128 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.704111 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.704353 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.704338 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.704553 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.704532 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.709688 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.709660 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69def90a-99a6-4c50-a5a0-a9182330ff80-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.710209 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.710189 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqfs\" (UniqueName: \"kubernetes.io/projected/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-kube-api-access-rbqfs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.710448 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.710429 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7ph\" (UniqueName: \"kubernetes.io/projected/69def90a-99a6-4c50-a5a0-a9182330ff80-kube-api-access-gc7ph\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndnrw\" (UID: \"69def90a-99a6-4c50-a5a0-a9182330ff80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.714112 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.714089 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688136b4-26ad-4cde-9bc8-c40ef5d7c2fa-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bjr2f\" (UID: \"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:02.843706 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.843626 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:02.885444 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:02.885413 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:03.124307 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.124280 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw"] Oct 30 19:59:03.125768 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:59:03.125745 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69def90a_99a6_4c50_a5a0_a9182330ff80.slice/crio-f1b78d71725443a1c28a0463878f7c17c6d3438554fec9d8b359515e590622b3 WatchSource:0}: Error finding container f1b78d71725443a1c28a0463878f7c17c6d3438554fec9d8b359515e590622b3: Status 404 returned error can't find the container with id f1b78d71725443a1c28a0463878f7c17c6d3438554fec9d8b359515e590622b3 Oct 30 19:59:03.128023 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.127991 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:03.128119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.128052 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:03.128119 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.128082 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:03.180415 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.180392 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f"] Oct 30 19:59:03.183512 ip-10-0-138-233 kubenswrapper[6514]: W1030 19:59:03.183486 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688136b4_26ad_4cde_9bc8_c40ef5d7c2fa.slice/crio-257b75ef3e088cf50c0227e3d1be541f2bf83a60e483cfe6fa7613a378e0890f WatchSource:0}: Error finding container 257b75ef3e088cf50c0227e3d1be541f2bf83a60e483cfe6fa7613a378e0890f: Status 404 returned error can't find the container with id 257b75ef3e088cf50c0227e3d1be541f2bf83a60e483cfe6fa7613a378e0890f Oct 30 19:59:03.185770 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.185736 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:03.185949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.185815 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:03.185949 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.185907 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:03.430076 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.430033 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" event={"ID":"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa","Type":"ContainerStarted","Data":"014204c3303351f7377d5e8dbd5cf2e412ccdb1cf7a0e0581d6c824612a0296f"} Oct 30 19:59:03.430269 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.430106 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" event={"ID":"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa","Type":"ContainerStarted","Data":"257b75ef3e088cf50c0227e3d1be541f2bf83a60e483cfe6fa7613a378e0890f"} Oct 30 19:59:03.431874 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.431832 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" event={"ID":"69def90a-99a6-4c50-a5a0-a9182330ff80","Type":"ContainerStarted","Data":"745dd97eb85345bcb7a028d96218c4a9d8fee4c52ca07c373e6b2924c356120e"} Oct 30 19:59:03.431990 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.431902 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" event={"ID":"69def90a-99a6-4c50-a5a0-a9182330ff80","Type":"ContainerStarted","Data":"f1b78d71725443a1c28a0463878f7c17c6d3438554fec9d8b359515e590622b3"} Oct 30 19:59:03.452117 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.452057 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podStartSLOduration=1.4520384210000001 podStartE2EDuration="1.452038421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:03.451370759 +0000 UTC m=+2513.409189948" watchObservedRunningTime="2026-10-30 19:59:03.452038421 +0000 UTC m=+2513.409857611" Oct 30 19:59:03.471878 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.471816 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podStartSLOduration=1.471799185 podStartE2EDuration="1.471799185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:03.470838978 +0000 UTC m=+2513.428658169" watchObservedRunningTime="2026-10-30 19:59:03.471799185 +0000 UTC m=+2513.429618377" Oct 30 19:59:03.844030 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.843949 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:03.885816 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:03.885791 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:04.849522 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:04.849488 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:04.891618 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:04.891585 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:05.442186 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:05.442148 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:05.442361 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:05.442342 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:05.443284 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:05.443259 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 19:59:05.443434 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:05.443315 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 19:59:13.365070 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:13.365041 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.880230 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:59:20.880193 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:59:26.203771 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:26.203688 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.887935 ip-10-0-138-233 kubenswrapper[6514]: E1030 19:59:50.887896 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 19:59:51.555549 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:51.555509 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 19:59:55.967610 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.967571 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6"] Oct 30 19:59:55.980277 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.980249 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:55.980791 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.980758 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6"] Oct 30 19:59:55.984797 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.984776 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:55.987655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.986454 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:59:55.987655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.986499 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:59:55.987655 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:55.986626 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:59:56.059499 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.059444 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9ph\" (UniqueName: \"kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.060575 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.060545 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.066999 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.066969 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.170385 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.170346 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9ph\" (UniqueName: \"kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.170582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.170403 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.170582 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.170457 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.171409 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.171378 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.173407 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.173380 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.179843 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.179812 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9ph\" (UniqueName: \"kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph\") pod \"maas-api-revoke-sub-d576e68b-crwz6\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.332166 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.332066 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 19:59:56.770458 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:56.770421 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6"] Oct 30 19:59:57.772251 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:57.772206 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" event={"ID":"19c0c871-76ab-4d28-980e-71b7db5209ed","Type":"ContainerStarted","Data":"50f8c56456d73a9c9a0ae71e33c89b8e8ed2bb1fe722c9b2835e6958c47262b2"} Oct 30 19:59:57.772780 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:57.772308 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" event={"ID":"19c0c871-76ab-4d28-980e-71b7db5209ed","Type":"ContainerStarted","Data":"df647a3b798a2f9fd078448690dc3e21482df2284eafed442e24d134d4bb50c2"} Oct 30 19:59:57.798685 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:57.798539 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" podStartSLOduration=2.798519703 podStartE2EDuration="2.798519703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:57.795901063 +0000 UTC m=+2567.753720253" watchObservedRunningTime="2026-10-30 19:59:57.798519703 +0000 UTC m=+2567.756338893" Oct 30 19:59:58.780926 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:58.780854 6514 generic.go:356] "Generic (PLEG): container finished" podID="19c0c871-76ab-4d28-980e-71b7db5209ed" containerID="50f8c56456d73a9c9a0ae71e33c89b8e8ed2bb1fe722c9b2835e6958c47262b2" exitCode=0 Oct 30 19:59:58.781308 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:58.780944 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" event={"ID":"19c0c871-76ab-4d28-980e-71b7db5209ed","Type":"ContainerDied","Data":"50f8c56456d73a9c9a0ae71e33c89b8e8ed2bb1fe722c9b2835e6958c47262b2"} Oct 30 19:59:59.977792 ip-10-0-138-233 kubenswrapper[6514]: I1030 19:59:59.977759 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 20:00:00.036531 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.036489 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca\") pod \"19c0c871-76ab-4d28-980e-71b7db5209ed\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " Oct 30 20:00:00.036773 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.036633 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca\") pod \"19c0c871-76ab-4d28-980e-71b7db5209ed\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " Oct 30 20:00:00.036843 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.036831 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph\" (UniqueName: \"kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph\") pod \"19c0c871-76ab-4d28-980e-71b7db5209ed\" (UID: \"19c0c871-76ab-4d28-980e-71b7db5209ed\") " Oct 30 20:00:00.038018 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.037988 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca" pod "19c0c871-76ab-4d28-980e-71b7db5209ed" (UID: "19c0c871-76ab-4d28-980e-71b7db5209ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:00.039851 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.039780 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph" pod "19c0c871-76ab-4d28-980e-71b7db5209ed" (UID: "19c0c871-76ab-4d28-980e-71b7db5209ed"). InnerVolumeSpecName "kube-api-access-qk9ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:00.040145 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.040116 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca" pod "19c0c871-76ab-4d28-980e-71b7db5209ed" (UID: "19c0c871-76ab-4d28-980e-71b7db5209ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:00.132459 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.132415 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp"] Oct 30 20:00:00.133188 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.133168 6514 cpu_manager.go:414] "containerMap: removing container" podUID="19c0c871-76ab-4d28-980e-71b7db5209ed" containerName="revoke-keys" Oct 30 20:00:00.133253 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.133193 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c0c871-76ab-4d28-980e-71b7db5209ed" containerName="revoke-keys" Oct 30 20:00:00.133391 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.133372 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c0c871-76ab-4d28-980e-71b7db5209ed" containerName="revoke-keys" Oct 30 20:00:00.138616 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.138577 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.138616 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.138616 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c0c871-76ab-4d28-980e-71b7db5209ed-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.138851 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.138636 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk9ph\" (UniqueName: \"kubernetes.io/projected/19c0c871-76ab-4d28-980e-71b7db5209ed-kube-api-access-qk9ph\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.146180 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.141371 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.148567 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.148528 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp"] Oct 30 20:00:00.239356 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.239316 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-hc2bp\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.239613 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.239583 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz\") pod \"maas-api-key-cleanup-29889840-hc2bp\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.341261 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.341164 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz\") pod \"maas-api-key-cleanup-29889840-hc2bp\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.341261 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.341218 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-hc2bp\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.342122 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.342092 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-hc2bp\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.350702 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.350672 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz\") pod \"maas-api-key-cleanup-29889840-hc2bp\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.473116 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.473073 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:00.786623 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.786590 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp"] Oct 30 20:00:00.789532 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:00.789498 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c426e72_40af_49d0_96af_03ed8e1ca863.slice/crio-14eea68bd4dcd496b882e8da66a20bdf7ce16d086c3b997d16fc8de271dde75d WatchSource:0}: Error finding container 14eea68bd4dcd496b882e8da66a20bdf7ce16d086c3b997d16fc8de271dde75d: Status 404 returned error can't find the container with id 14eea68bd4dcd496b882e8da66a20bdf7ce16d086c3b997d16fc8de271dde75d Oct 30 20:00:00.795884 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.795830 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" event={"ID":"19c0c871-76ab-4d28-980e-71b7db5209ed","Type":"ContainerDied","Data":"df647a3b798a2f9fd078448690dc3e21482df2284eafed442e24d134d4bb50c2"} Oct 30 20:00:00.795884 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.795877 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6" Oct 30 20:00:00.796074 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:00.795909 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df647a3b798a2f9fd078448690dc3e21482df2284eafed442e24d134d4bb50c2" Oct 30 20:00:01.803890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:01.802781 6514 generic.go:356] "Generic (PLEG): container finished" podID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerID="4d9bc13ec84a16f8bcff3497984fc51766ac955ed591fc3fafa4c2d4f62978ea" exitCode=60 Oct 30 20:00:01.803890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:01.802829 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" event={"ID":"0c426e72-40af-49d0-96af-03ed8e1ca863","Type":"ContainerDied","Data":"4d9bc13ec84a16f8bcff3497984fc51766ac955ed591fc3fafa4c2d4f62978ea"} Oct 30 20:00:01.803890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:01.802905 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" event={"ID":"0c426e72-40af-49d0-96af-03ed8e1ca863","Type":"ContainerStarted","Data":"14eea68bd4dcd496b882e8da66a20bdf7ce16d086c3b997d16fc8de271dde75d"} Oct 30 20:00:01.803890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:01.803279 6514 scope.go:122] "RemoveContainer" containerID="4d9bc13ec84a16f8bcff3497984fc51766ac955ed591fc3fafa4c2d4f62978ea" Oct 30 20:00:02.809940 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:02.809904 6514 generic.go:356] "Generic (PLEG): container finished" podID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerID="5bc2322a3a86cbc82a1b1ea3ab8b00bf1c66ab494ace0e1ce2abc8d7c1650d53" exitCode=60 Oct 30 20:00:02.810398 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:02.809961 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" event={"ID":"0c426e72-40af-49d0-96af-03ed8e1ca863","Type":"ContainerDied","Data":"5bc2322a3a86cbc82a1b1ea3ab8b00bf1c66ab494ace0e1ce2abc8d7c1650d53"} Oct 30 20:00:02.810398 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:02.810032 6514 scope.go:122] "RemoveContainer" containerID="4d9bc13ec84a16f8bcff3497984fc51766ac955ed591fc3fafa4c2d4f62978ea" Oct 30 20:00:02.810795 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:02.810769 6514 scope.go:122] "RemoveContainer" containerID="5bc2322a3a86cbc82a1b1ea3ab8b00bf1c66ab494ace0e1ce2abc8d7c1650d53" Oct 30 20:00:02.811358 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:00:02.811098 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-hc2bp_odh-ai-gateway-infra(0c426e72-40af-49d0-96af-03ed8e1ca863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" Oct 30 20:00:03.815823 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.815786 6514 generic.go:356] "Generic (PLEG): container finished" podID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerID="745dd97eb85345bcb7a028d96218c4a9d8fee4c52ca07c373e6b2924c356120e" exitCode=0 Oct 30 20:00:03.816331 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.815857 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" event={"ID":"69def90a-99a6-4c50-a5a0-a9182330ff80","Type":"ContainerDied","Data":"745dd97eb85345bcb7a028d96218c4a9d8fee4c52ca07c373e6b2924c356120e"} Oct 30 20:00:03.816331 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.816301 6514 scope.go:122] "RemoveContainer" containerID="745dd97eb85345bcb7a028d96218c4a9d8fee4c52ca07c373e6b2924c356120e" Oct 30 20:00:03.817619 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.817581 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.817897 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.817699 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.817897 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.817746 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.817897 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.817761 6514 scope.go:122] "RemoveContainer" containerID="5bc2322a3a86cbc82a1b1ea3ab8b00bf1c66ab494ace0e1ce2abc8d7c1650d53" Oct 30 20:00:03.818115 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:00:03.818023 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-hc2bp_odh-ai-gateway-infra(0c426e72-40af-49d0-96af-03ed8e1ca863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" Oct 30 20:00:03.826640 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.826609 6514 generic.go:356] "Generic (PLEG): container finished" podID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerID="014204c3303351f7377d5e8dbd5cf2e412ccdb1cf7a0e0581d6c824612a0296f" exitCode=0 Oct 30 20:00:03.826776 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.826675 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" event={"ID":"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa","Type":"ContainerDied","Data":"014204c3303351f7377d5e8dbd5cf2e412ccdb1cf7a0e0581d6c824612a0296f"} Oct 30 20:00:03.827159 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.827141 6514 scope.go:122] "RemoveContainer" containerID="014204c3303351f7377d5e8dbd5cf2e412ccdb1cf7a0e0581d6c824612a0296f" Oct 30 20:00:03.829264 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.829242 6514 generic.go:356] "Generic (PLEG): container finished" podID="88856f6f-b914-4487-9da6-5107913dc38e" containerID="f9808cebfd76b62971877c65dd55c3d751c0d1b9c3cc8f9068f631fc082cf271" exitCode=0 Oct 30 20:00:03.829344 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.829309 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" event={"ID":"88856f6f-b914-4487-9da6-5107913dc38e","Type":"ContainerDied","Data":"f9808cebfd76b62971877c65dd55c3d751c0d1b9c3cc8f9068f631fc082cf271"} Oct 30 20:00:03.829592 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.829577 6514 scope.go:122] "RemoveContainer" containerID="f9808cebfd76b62971877c65dd55c3d751c0d1b9c3cc8f9068f631fc082cf271" Oct 30 20:00:03.831072 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.831046 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.831173 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.831115 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.831173 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.831148 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.831292 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.831188 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.831292 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.831280 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:03.831407 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:03.831364 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:04.862887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:04.862222 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" event={"ID":"88856f6f-b914-4487-9da6-5107913dc38e","Type":"ContainerStarted","Data":"a386f199bd046b892efd2aa6b5dda6098da82be07ee4a2d9f20307503f3d5dbf"} Oct 30 20:00:04.871483 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:04.871451 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" event={"ID":"69def90a-99a6-4c50-a5a0-a9182330ff80","Type":"ContainerStarted","Data":"b17c53945b60b439df7b96ecf161d5290fd265739283add36ac3f1f67123ef84"} Oct 30 20:00:04.873467 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:04.873442 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" event={"ID":"688136b4-26ad-4cde-9bc8-c40ef5d7c2fa","Type":"ContainerStarted","Data":"b9ef6e6b1af8666e23f2e900f3ea0c988677659caca3ba41974a4b758c6a35f2"} Oct 30 20:00:04.885904 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:04.885881 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 20:00:04.887328 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:04.887303 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:04.887440 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:04.887348 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:05.365892 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.365301 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 20:00:05.454890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.444962 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 20:00:05.454890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.445067 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 20:00:05.858151 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.858118 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 20:00:05.866986 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.866954 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 20:00:05.867577 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.863977 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:00:05.867752 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.867727 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 20:00:05.868136 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.867941 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:00:05.868311 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.868289 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 20:00:05.886658 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.886629 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:05.886788 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:05.886676 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:06.845889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:06.845255 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:00:06.845889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:06.845315 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 20:00:06.859887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:06.854702 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:00:06.859887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:06.854761 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 20:00:06.886692 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:06.886660 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:06.887244 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:06.886703 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:07.264999 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.262787 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:00:07.848329 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.848288 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:00:07.848635 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.848607 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 20:00:07.857250 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.857219 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:00:07.858588 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.858558 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 20:00:07.888280 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.888252 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:07.888848 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:07.888801 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:08.847887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:08.847330 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:00:08.847887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:08.847400 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 20:00:08.857889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:08.856226 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:00:08.857889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:08.856278 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 20:00:08.885941 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:08.885906 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:08.886090 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:08.885956 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:09.848889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:09.846266 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:00:09.848889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:09.846330 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 20:00:09.856889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:09.855507 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:00:09.856889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:09.855558 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 20:00:09.886761 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:09.886725 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:09.886950 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:09.886783 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:10.845212 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:10.845167 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:00:10.845458 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:10.845243 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 20:00:10.855327 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:10.855291 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:00:10.855958 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:10.855925 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 20:00:10.886442 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:10.886403 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 20:00:10.886643 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:10.886466 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 20:00:11.466065 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:11.466025 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:00:12.332890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.331985 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:12.332890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.332088 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:12.858480 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.858439 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:12.858824 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.858778 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:12.887064 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.887010 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:12.887270 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.887087 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:12.928075 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.928016 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:12.928285 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:12.928097 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:13.152561 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.152458 6514 scope.go:122] "RemoveContainer" containerID="5bc2322a3a86cbc82a1b1ea3ab8b00bf1c66ab494ace0e1ce2abc8d7c1650d53" Oct 30 20:00:13.468539 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.468509 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:13.887565 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.887521 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:13.887793 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.887586 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:13.901034 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.900988 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:13.901197 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.901041 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:13.929644 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.929610 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:13.929806 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:13.929663 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:14.013326 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.013283 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" event={"ID":"0c426e72-40af-49d0-96af-03ed8e1ca863","Type":"ContainerStarted","Data":"bf235efe75cf7f7363c0d4bd02ca23aca55e0c586f873f1b9237eece055898a6"} Oct 30 20:00:14.058233 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.058174 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" podStartSLOduration=14.05815579 podStartE2EDuration="14.05815579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:14.053666246 +0000 UTC m=+2584.011485437" watchObservedRunningTime="2026-10-30 20:00:14.05815579 +0000 UTC m=+2584.015974981" Oct 30 20:00:14.930887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.930524 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:14.930887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.930594 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:14.954601 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.953955 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:14.954601 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.954016 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:14.973030 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.972982 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:14.973224 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:14.973038 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:15.056187 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.056147 6514 generic.go:356] "Generic (PLEG): container finished" podID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerID="bf235efe75cf7f7363c0d4bd02ca23aca55e0c586f873f1b9237eece055898a6" exitCode=60 Oct 30 20:00:15.056542 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.056509 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" event={"ID":"0c426e72-40af-49d0-96af-03ed8e1ca863","Type":"ContainerDied","Data":"bf235efe75cf7f7363c0d4bd02ca23aca55e0c586f873f1b9237eece055898a6"} Oct 30 20:00:15.056736 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.056719 6514 scope.go:122] "RemoveContainer" containerID="5bc2322a3a86cbc82a1b1ea3ab8b00bf1c66ab494ace0e1ce2abc8d7c1650d53" Oct 30 20:00:15.057350 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.057329 6514 scope.go:122] "RemoveContainer" containerID="bf235efe75cf7f7363c0d4bd02ca23aca55e0c586f873f1b9237eece055898a6" Oct 30 20:00:15.057779 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:00:15.057754 6514 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-hc2bp_odh-ai-gateway-infra(0c426e72-40af-49d0-96af-03ed8e1ca863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" Oct 30 20:00:15.065888 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.064150 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp"] Oct 30 20:00:15.931169 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.931136 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:15.931714 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.931187 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:15.954553 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.954511 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:15.954711 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.954571 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:15.973375 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.973327 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:15.973530 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:15.973381 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:16.221577 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.221550 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:16.324937 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.324902 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca\") pod \"0c426e72-40af-49d0-96af-03ed8e1ca863\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " Oct 30 20:00:16.325155 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.325014 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz\") pod \"0c426e72-40af-49d0-96af-03ed8e1ca863\" (UID: \"0c426e72-40af-49d0-96af-03ed8e1ca863\") " Oct 30 20:00:16.325363 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.325332 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca" pod "0c426e72-40af-49d0-96af-03ed8e1ca863" (UID: "0c426e72-40af-49d0-96af-03ed8e1ca863"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:16.327975 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.327947 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz" pod "0c426e72-40af-49d0-96af-03ed8e1ca863" (UID: "0c426e72-40af-49d0-96af-03ed8e1ca863"). InnerVolumeSpecName "kube-api-access-vsvsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:16.426754 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.426712 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c426e72-40af-49d0-96af-03ed8e1ca863-maas-api-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:16.426754 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.426746 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsvsz\" (UniqueName: \"kubernetes.io/projected/0c426e72-40af-49d0-96af-03ed8e1ca863-kube-api-access-vsvsz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:16.973100 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.973025 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:16.973610 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.973558 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:16.998971 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.998943 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:16.999181 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:16.999156 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:17.016884 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.014966 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:17.016884 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.015011 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:17.073283 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.073250 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" event={"ID":"0c426e72-40af-49d0-96af-03ed8e1ca863","Type":"ContainerDied","Data":"14eea68bd4dcd496b882e8da66a20bdf7ce16d086c3b997d16fc8de271dde75d"} Oct 30 20:00:17.073455 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.073307 6514 scope.go:122] "RemoveContainer" containerID="bf235efe75cf7f7363c0d4bd02ca23aca55e0c586f873f1b9237eece055898a6" Oct 30 20:00:17.073455 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.073388 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp" Oct 30 20:00:17.114899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.102715 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp"] Oct 30 20:00:17.114899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.109652 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-hc2bp"] Oct 30 20:00:17.236936 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.216692 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7"] Oct 30 20:00:17.237911 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.237842 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.238096 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.238079 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.238222 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.238208 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.238375 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.238343 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.238512 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.238498 6514 cpu_manager.go:414] "containerMap: removing container" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.238610 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.238597 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.241884 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.238787 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.242039 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.242020 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.242152 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.242139 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" containerName="cleanup" Oct 30 20:00:17.268260 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.268233 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7"] Oct 30 20:00:17.268546 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.268493 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.277651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.277606 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 20:00:17.277968 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.277952 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:17.278186 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.278174 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:00:17.278400 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.278333 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:00:17.347539 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.347500 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.347888 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.347838 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.348132 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.348111 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdjv\" (UniqueName: \"kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.458518 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.458473 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdjv\" (UniqueName: \"kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.459176 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.459148 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.459422 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.459400 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.460851 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.460817 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.470471 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.470448 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.472777 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.472754 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdjv\" (UniqueName: \"kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv\") pod \"maas-api-revoke-sub-d2bc93bc-grvd7\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:17.590325 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:17.590249 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:18.017010 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.016963 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:18.017524 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.017053 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:18.057992 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.011850 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7"] Oct 30 20:00:18.058358 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.058331 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-82vrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:18.058602 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.058572 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" podUID="88856f6f-b914-4487-9da6-5107913dc38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:18.059987 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.059945 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:18.060145 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.060127 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:18.062961 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:18.062889 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0db0ca3_0b9b_4066_a97e_95565077a1d1.slice/crio-2e722446375fed75d0a686f4f5d3218eb1c28d83db2eafa74bad2a25383eb5c0 WatchSource:0}: Error finding container 2e722446375fed75d0a686f4f5d3218eb1c28d83db2eafa74bad2a25383eb5c0: Status 404 returned error can't find the container with id 2e722446375fed75d0a686f4f5d3218eb1c28d83db2eafa74bad2a25383eb5c0 Oct 30 20:00:18.082298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.082274 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" event={"ID":"a0db0ca3-0b9b-4066-a97e-95565077a1d1","Type":"ContainerStarted","Data":"2e722446375fed75d0a686f4f5d3218eb1c28d83db2eafa74bad2a25383eb5c0"} Oct 30 20:00:18.167308 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.167274 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c426e72-40af-49d0-96af-03ed8e1ca863" path="/var/lib/kubelet/pods/0c426e72-40af-49d0-96af-03ed8e1ca863/volumes" Oct 30 20:00:18.183814 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.183776 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 20:00:18.184927 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.184905 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-82vrf" Oct 30 20:00:18.489587 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.489555 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 20:00:18.490716 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.490663 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" Oct 30 20:00:18.669015 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.668918 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 20:00:18.670352 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:18.670329 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" Oct 30 20:00:19.089089 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:19.089063 6514 generic.go:356] "Generic (PLEG): container finished" podID="a0db0ca3-0b9b-4066-a97e-95565077a1d1" containerID="4a5d56e52c862943c91371dd231abd1b4635ee655d87bce89d097d78c20553b9" exitCode=0 Oct 30 20:00:19.089446 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:19.089161 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" event={"ID":"a0db0ca3-0b9b-4066-a97e-95565077a1d1","Type":"ContainerDied","Data":"4a5d56e52c862943c91371dd231abd1b4635ee655d87bce89d097d78c20553b9"} Oct 30 20:00:20.223492 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.223465 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:20.399552 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.399471 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca\") pod \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " Oct 30 20:00:20.399725 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.399667 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca\") pod \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " Oct 30 20:00:20.399725 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.399709 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv\" (UniqueName: \"kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv\") pod \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\" (UID: \"a0db0ca3-0b9b-4066-a97e-95565077a1d1\") " Oct 30 20:00:20.400037 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.400014 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca" pod "a0db0ca3-0b9b-4066-a97e-95565077a1d1" (UID: "a0db0ca3-0b9b-4066-a97e-95565077a1d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.401617 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.401593 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca" pod "a0db0ca3-0b9b-4066-a97e-95565077a1d1" (UID: "a0db0ca3-0b9b-4066-a97e-95565077a1d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:20.401733 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.401678 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv" pod "a0db0ca3-0b9b-4066-a97e-95565077a1d1" (UID: "a0db0ca3-0b9b-4066-a97e-95565077a1d1"). InnerVolumeSpecName "kube-api-access-tgdjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.500735 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.500708 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.500735 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.500730 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0ca3-0b9b-4066-a97e-95565077a1d1-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.500916 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:20.500741 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdjv\" (UniqueName: \"kubernetes.io/projected/a0db0ca3-0b9b-4066-a97e-95565077a1d1-kube-api-access-tgdjv\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.881876 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:00:20.881820 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:00:21.108339 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:21.108298 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" event={"ID":"a0db0ca3-0b9b-4066-a97e-95565077a1d1","Type":"ContainerDied","Data":"2e722446375fed75d0a686f4f5d3218eb1c28d83db2eafa74bad2a25383eb5c0"} Oct 30 20:00:21.108521 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:21.108380 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e722446375fed75d0a686f4f5d3218eb1c28d83db2eafa74bad2a25383eb5c0" Oct 30 20:00:21.108521 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:21.108317 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7" Oct 30 20:00:28.148121 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:28.148090 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:00:28.895732 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:28.895681 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.262931 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.262894 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp"] Oct 30 20:00:29.263345 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.263322 6514 cpu_manager.go:414] "containerMap: removing container" podUID="a0db0ca3-0b9b-4066-a97e-95565077a1d1" containerName="revoke-keys" Oct 30 20:00:29.263345 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.263336 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0db0ca3-0b9b-4066-a97e-95565077a1d1" containerName="revoke-keys" Oct 30 20:00:29.263449 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.263418 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0db0ca3-0b9b-4066-a97e-95565077a1d1" containerName="revoke-keys" Oct 30 20:00:29.265698 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.265671 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.271619 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.271594 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-cb7041-openshift-default-dockercfg-9hrfr\"" Oct 30 20:00:29.277131 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.277101 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp"] Oct 30 20:00:29.284643 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.284621 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.284742 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.284648 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvsj9\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.284742 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.284688 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.284833 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.284764 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.284833 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.284815 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.284943 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.284878 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.285031 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.285013 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.285070 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.285046 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.285108 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.285076 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.323391 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.323363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.386041 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386009 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386053 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386081 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386118 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386193 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386220 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386249 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386580 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386303 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386580 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386329 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvsj9\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386702 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386625 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.386952 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.386908 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.387257 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.387232 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.387406 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.387259 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.387585 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.387561 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.389041 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.389015 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.389235 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.389217 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.395136 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.395107 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvsj9\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.395675 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.395653 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token\") pod \"e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.432803 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.432773 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.583275 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.583189 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:29.863048 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.863010 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp"] Oct 30 20:00:29.864780 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:29.864752 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53800085_0fac_45e9_a254_7c2a0776eb7d.slice/crio-957ab10e818a381db288824525184cf808a0c33917e8f710ab781d1c37a46f7b WatchSource:0}: Error finding container 957ab10e818a381db288824525184cf808a0c33917e8f710ab781d1c37a46f7b: Status 404 returned error can't find the container with id 957ab10e818a381db288824525184cf808a0c33917e8f710ab781d1c37a46f7b Oct 30 20:00:29.867300 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.867260 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:29.867381 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.867355 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:29.867427 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:29.867401 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:30.158235 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:30.158161 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" event={"ID":"53800085-0fac-45e9-a254-7c2a0776eb7d","Type":"ContainerStarted","Data":"a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68"} Oct 30 20:00:30.158235 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:30.158211 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" event={"ID":"53800085-0fac-45e9-a254-7c2a0776eb7d","Type":"ContainerStarted","Data":"957ab10e818a381db288824525184cf808a0c33917e8f710ab781d1c37a46f7b"} Oct 30 20:00:30.178187 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:30.178139 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podStartSLOduration=1.17812547 podStartE2EDuration="1.17812547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:30.175382351 +0000 UTC m=+2600.133201541" watchObservedRunningTime="2026-10-30 20:00:30.17812547 +0000 UTC m=+2600.135944659" Oct 30 20:00:30.583887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:30.583836 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:30.588950 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:30.588924 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:31.158196 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:31.158163 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:31.159096 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:31.159074 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:32.420412 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:32.420370 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:32.585809 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:32.585757 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:33.627488 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.627456 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd"] Oct 30 20:00:33.630543 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.630520 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.637168 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.637139 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-cb7041-serving-cert\"" Oct 30 20:00:33.639083 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.639049 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2z7lq\"" Oct 30 20:00:33.639083 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.639069 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:00:33.639229 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.639137 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:00:33.640515 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.640494 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:00:33.646845 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.646810 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd"] Oct 30 20:00:33.657045 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.657020 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:33.692515 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.692484 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67"] Oct 30 20:00:33.695260 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.695238 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.707797 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.707771 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67"] Oct 30 20:00:33.731141 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.731110 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j"] Oct 30 20:00:33.732704 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.732678 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.732838 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.732818 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.732923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.732905 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.733019 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.732993 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kclkb\" (UniqueName: \"kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb\") pod \"payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.733078 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.733042 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.733078 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.733069 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqxw\" (UniqueName: \"kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.733834 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.733815 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:33.735131 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.735110 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-abafe0b0-dockercfg-9p5zn\"" Oct 30 20:00:33.737825 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.737807 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-cb7041\"" Oct 30 20:00:33.745714 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.745692 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j"] Oct 30 20:00:33.820272 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.820202 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:33.833639 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.833605 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:33.833800 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.833653 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.833800 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.833729 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.833800 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.833774 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldxq\" (UniqueName: \"kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq\") pod \"payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:33.834024 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.833892 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kclkb\" (UniqueName: \"kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb\") pod \"payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.834024 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.833963 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.834024 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.834009 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqxw\" (UniqueName: \"kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.834190 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.834064 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.834729 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.834703 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.834890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.834818 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.836618 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.836589 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.837529 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.837502 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.850515 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.850493 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqxw\" (UniqueName: \"kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw\") pod \"maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.851053 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.851023 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kclkb\" (UniqueName: \"kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb\") pod \"payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:33.935518 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.935484 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:33.935699 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.935556 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldxq\" (UniqueName: \"kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq\") pod \"payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:33.936221 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.936199 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:33.945324 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.945298 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:33.950501 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:33.950479 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldxq\" (UniqueName: \"kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq\") pod \"payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:34.009402 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:34.008300 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:34.047150 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:34.047118 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:34.252552 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:34.252520 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd"] Oct 30 20:00:34.254058 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:34.254033 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694f1cf0_fc83_4094_a622_55c80e872163.slice/crio-4e568ad5bc39b1e77640a461cccb7720618850ef0aa20f0a59cc17a43cae4d6d WatchSource:0}: Error finding container 4e568ad5bc39b1e77640a461cccb7720618850ef0aa20f0a59cc17a43cae4d6d: Status 404 returned error can't find the container with id 4e568ad5bc39b1e77640a461cccb7720618850ef0aa20f0a59cc17a43cae4d6d Oct 30 20:00:34.306001 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:34.305964 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67"] Oct 30 20:00:34.309185 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:34.309149 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c501f7_ed94_4a95_8881_a4153f55bb2e.slice/crio-c68eb662693ec3f3e2a3eda18572941e8dc246b8fc744e269ccdc5a25161e494 WatchSource:0}: Error finding container c68eb662693ec3f3e2a3eda18572941e8dc246b8fc744e269ccdc5a25161e494: Status 404 returned error can't find the container with id c68eb662693ec3f3e2a3eda18572941e8dc246b8fc744e269ccdc5a25161e494 Oct 30 20:00:34.373703 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:34.373675 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j"] Oct 30 20:00:34.380649 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:34.380622 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec72bd20_3199_4b14_b963_4ff03634ae0e.slice/crio-e46c219ecbde3dbc350137e60718fb31f4335013260747b45618df80f28eecee WatchSource:0}: Error finding container e46c219ecbde3dbc350137e60718fb31f4335013260747b45618df80f28eecee: Status 404 returned error can't find the container with id e46c219ecbde3dbc350137e60718fb31f4335013260747b45618df80f28eecee Oct 30 20:00:34.901103 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:34.901054 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.178391 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.178301 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" event={"ID":"694f1cf0-fc83-4094-a622-55c80e872163","Type":"ContainerStarted","Data":"138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48"} Oct 30 20:00:35.178391 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.178359 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" event={"ID":"694f1cf0-fc83-4094-a622-55c80e872163","Type":"ContainerStarted","Data":"4e568ad5bc39b1e77640a461cccb7720618850ef0aa20f0a59cc17a43cae4d6d"} Oct 30 20:00:35.178391 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.178390 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:35.180054 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.180024 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" event={"ID":"ec72bd20-3199-4b14-b963-4ff03634ae0e","Type":"ContainerStarted","Data":"e30fb61cae867eff82958f8da649bb699eb6b74794e2b1d352fb1592355ed015"} Oct 30 20:00:35.180185 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.180078 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" event={"ID":"ec72bd20-3199-4b14-b963-4ff03634ae0e","Type":"ContainerStarted","Data":"e46c219ecbde3dbc350137e60718fb31f4335013260747b45618df80f28eecee"} Oct 30 20:00:35.180185 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.180131 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:35.181471 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.181444 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" event={"ID":"05c501f7-ed94-4a95-8881-a4153f55bb2e","Type":"ContainerStarted","Data":"5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0"} Oct 30 20:00:35.181627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.181485 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" event={"ID":"05c501f7-ed94-4a95-8881-a4153f55bb2e","Type":"ContainerStarted","Data":"c68eb662693ec3f3e2a3eda18572941e8dc246b8fc744e269ccdc5a25161e494"} Oct 30 20:00:35.181627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.181516 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:35.237668 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.237624 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" podStartSLOduration=1.775124703 podStartE2EDuration="2.237612846s" totalImagesPullingTime="462.488143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:35.212836754 +0000 UTC m=+2605.170655953" watchObservedRunningTime="2026-10-30 20:00:35.237612846 +0000 UTC m=+2605.195432035" Oct 30 20:00:35.285283 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.285226 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" podStartSLOduration=2.2852109130000002 podStartE2EDuration="2.285210913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:35.283202253 +0000 UTC m=+2605.241021442" watchObservedRunningTime="2026-10-30 20:00:35.285210913 +0000 UTC m=+2605.243030103" Oct 30 20:00:35.286173 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.286135 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" podStartSLOduration=2.286123744 podStartE2EDuration="2.286123744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:35.240553184 +0000 UTC m=+2605.198372373" watchObservedRunningTime="2026-10-30 20:00:35.286123744 +0000 UTC m=+2605.243942935" Oct 30 20:00:35.778000 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:35.777963 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.082069 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:36.081987 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:00:36.681013 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:36.680975 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:37.473575 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:37.473545 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.173217 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:38.173185 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.072527 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:39.072489 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.920342 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:39.920303 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:40.660566 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:40.660525 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:41.215890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:41.208020 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:00:41.483440 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:41.483355 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:42.252533 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:42.252501 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:42.992812 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:42.992773 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.888419 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:43.888390 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.652323 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:44.652271 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.406806 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:45.406777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.156073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.156033 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.160199 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.160173 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 20:00:46.160328 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.160238 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 20:00:46.189541 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.189500 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:00:46.190253 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.190226 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:00:46.270977 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.270945 6514 generic.go:356] "Generic (PLEG): container finished" podID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerID="a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68" exitCode=0 Oct 30 20:00:46.271171 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.271029 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" event={"ID":"53800085-0fac-45e9-a254-7c2a0776eb7d","Type":"ContainerDied","Data":"a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68"} Oct 30 20:00:46.271437 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.271420 6514 scope.go:122] "RemoveContainer" containerID="a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68" Oct 30 20:00:46.272885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.272826 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:46.272979 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.272956 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:46.273071 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.273052 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:46.968153 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:46.968117 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.278148 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:47.278066 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" event={"ID":"53800085-0fac-45e9-a254-7c2a0776eb7d","Type":"ContainerStarted","Data":"5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92"} Oct 30 20:00:47.425481 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:47.425446 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.583986 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:47.583898 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:47.584163 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:47.584133 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 20:00:47.584249 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:47.584227 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 20:00:47.827645 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:47.827606 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.038981 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:48.038945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.169247 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:48.169219 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.282686 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:48.282655 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.583692 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:48.583660 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 20:00:48.583896 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:48.583727 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 20:00:48.761762 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:48.761718 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.516769 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.516731 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82"] Oct 30 20:00:49.519725 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.519704 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.530959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.530933 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82"] Oct 30 20:00:49.531828 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.531806 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-cb7041\"/\"kube-root-ca.crt\"" Oct 30 20:00:49.532433 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.532413 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-cb7041\"/\"openshift-service-ca.crt\"" Oct 30 20:00:49.536374 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.536352 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.536475 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.536396 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.536519 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.536500 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.536557 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.536530 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.536557 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.536550 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lg5\" (UniqueName: \"kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.536661 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.536648 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.575026 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.575004 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.584129 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.584106 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 20:00:49.584218 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.584148 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 20:00:49.613184 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.613161 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-cb7041\"/\"default-dockercfg-ssx4j\"" Oct 30 20:00:49.614619 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.614557 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-cb7041\"/\"simulator-single-main-cb7041-kserve-self-signed-certs\"" Oct 30 20:00:49.637725 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.637694 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.637908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.637742 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.637908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.637777 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lg5\" (UniqueName: \"kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.637908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.637826 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.638034 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.637927 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.638034 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.637978 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.638636 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.638332 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.638636 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.638547 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.638636 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.638587 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.640746 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.640723 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.641214 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.641189 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.655568 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.655540 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lg5\" (UniqueName: \"kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5\") pod \"simulator-single-main-cb7041-kserve-7c6677896c-t9g82\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:49.767655 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.767550 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.832548 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:49.832513 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:50.116073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.115981 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82"] Oct 30 20:00:50.120036 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:00:50.120003 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda60b9d9f_1294_4e00_a981_2c51fdc7e808.slice/crio-d0f3b8446c6c6cd3601d2b3260a48564313466e77f4c394fafccbd92e2d30855 WatchSource:0}: Error finding container d0f3b8446c6c6cd3601d2b3260a48564313466e77f4c394fafccbd92e2d30855: Status 404 returned error can't find the container with id d0f3b8446c6c6cd3601d2b3260a48564313466e77f4c394fafccbd92e2d30855 Oct 30 20:00:50.320728 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.320690 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" event={"ID":"a60b9d9f-1294-4e00-a981-2c51fdc7e808","Type":"ContainerStarted","Data":"59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc"} Oct 30 20:00:50.320728 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.320740 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" event={"ID":"a60b9d9f-1294-4e00-a981-2c51fdc7e808","Type":"ContainerStarted","Data":"d0f3b8446c6c6cd3601d2b3260a48564313466e77f4c394fafccbd92e2d30855"} Oct 30 20:00:50.339556 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.339508 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" podStartSLOduration=1.339496035 podStartE2EDuration="1.339496035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:50.338335091 +0000 UTC m=+2620.296154282" watchObservedRunningTime="2026-10-30 20:00:50.339496035 +0000 UTC m=+2620.297315224" Oct 30 20:00:50.543527 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.543478 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:50.583790 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.583755 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 20:00:50.583991 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.583812 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 20:00:50.833087 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:50.833007 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:50.881039 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:00:50.881001 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:00:51.137520 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:51.137426 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:00:51.367438 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:51.367406 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.584147 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:51.584112 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 20:00:51.584537 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:51.584166 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 20:00:52.252312 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:52.252206 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:52.725629 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:52.725597 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:52.726083 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:52.725893 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:52.726757 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:52.726738 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:00:52.951090 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:52.951061 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:53.742953 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:53.742922 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.523210 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:54.523177 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.159855 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:55.159821 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.857808 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:55.857761 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.648019 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:56.647975 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.575198 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:57.575163 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:58.452923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:58.452889 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.142050 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:59.142015 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.833108 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:59.833072 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:59.845487 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:59.845463 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:00:59.847052 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:00:59.847018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:00.384013 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:00.383980 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:01:00.661361 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:00.661275 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.531314 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:01.531283 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.254447 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:02.254419 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.940094 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:02.940047 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.642323 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:03.642286 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.340501 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:04.340446 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.154153 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:05.154120 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.844740 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:05.844712 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.637423 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:06.637384 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.563395 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:07.563354 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.857217 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:07.857124 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:01:08.476564 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:08.476527 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.259762 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:09.259729 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.030206 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:10.030172 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.728513 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:10.728473 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:11.442012 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:11.441985 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.155994 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:12.155956 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.861558 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:12.861523 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.492899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:13.492848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.613209 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:13.613134 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:14.262823 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:14.262790 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:14.956104 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:14.956071 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:15.635211 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:15.635174 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:16.346975 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:16.346942 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.097595 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:17.097554 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.920925 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:17.920885 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.660855 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:18.660822 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.347902 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:19.347848 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.053824 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:20.053784 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.751061 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:20.751023 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.881491 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:01:20.881450 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:01:21.652813 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:21.652778 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.352087 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:22.352055 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.143747 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:23.143711 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.935003 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:23.934968 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:24.680524 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:24.680470 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.545091 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:25.545061 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.252141 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:26.252105 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.955673 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:26.955639 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:27.661202 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:27.661160 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.361022 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:28.360978 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.148292 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:29.148258 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.854598 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:29.854559 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.361637 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:30.361604 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.718411 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:30.718379 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:01:30.729327 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:30.729301 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.592098 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:31.592065 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:32.332796 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:32.332761 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:33.260543 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:33.260500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.040513 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:34.040482 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.744877 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:34.744804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:35.300514 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:35.300484 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82"] Oct 30 20:01:35.396815 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:35.396781 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:35.500260 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:35.500228 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:36.306008 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:36.305969 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:36.936441 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:36.936404 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:37.652285 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:37.652256 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:38.348429 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:38.348395 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.061934 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:39.061896 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.743298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:39.743266 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.443228 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:40.443192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.681518 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:40.681476 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j"] Oct 30 20:01:40.697114 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:40.697035 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67"] Oct 30 20:01:40.769609 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:40.769568 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.871875 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:40.871834 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.147954 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:41.147854 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.670253 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:41.670220 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd"] Oct 30 20:01:41.670684 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:41.670438 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" podUID="694f1cf0-fc83-4094-a622-55c80e872163" containerName="maas-api" containerID="cri-o://138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48" gracePeriod=30 Oct 30 20:01:41.753410 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:41.753380 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.895992 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:41.895961 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.010150 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.010118 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.136366 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.136336 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:01:42.174930 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.174884 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs\") pod \"694f1cf0-fc83-4094-a622-55c80e872163\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " Oct 30 20:01:42.175096 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.174977 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw\" (UniqueName: \"kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw\") pod \"694f1cf0-fc83-4094-a622-55c80e872163\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " Oct 30 20:01:42.175096 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.175061 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca\") pod \"694f1cf0-fc83-4094-a622-55c80e872163\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " Oct 30 20:01:42.175183 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.175103 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls\") pod \"694f1cf0-fc83-4094-a622-55c80e872163\" (UID: \"694f1cf0-fc83-4094-a622-55c80e872163\") " Oct 30 20:01:42.175617 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.175582 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca" pod "694f1cf0-fc83-4094-a622-55c80e872163" (UID: "694f1cf0-fc83-4094-a622-55c80e872163"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:42.177298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.177271 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs" pod "694f1cf0-fc83-4094-a622-55c80e872163" (UID: "694f1cf0-fc83-4094-a622-55c80e872163"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:42.177583 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.177560 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw" pod "694f1cf0-fc83-4094-a622-55c80e872163" (UID: "694f1cf0-fc83-4094-a622-55c80e872163"). InnerVolumeSpecName "kube-api-access-fgqxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:42.177931 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.177903 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls" pod "694f1cf0-fc83-4094-a622-55c80e872163" (UID: "694f1cf0-fc83-4094-a622-55c80e872163"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:42.277127 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.277040 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694f1cf0-fc83-4094-a622-55c80e872163-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:42.277127 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.277081 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:42.277127 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.277097 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694f1cf0-fc83-4094-a622-55c80e872163-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:42.277127 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.277112 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqxw\" (UniqueName: \"kubernetes.io/projected/694f1cf0-fc83-4094-a622-55c80e872163-kube-api-access-fgqxw\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:42.464417 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.464363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.602651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.602553 6514 generic.go:356] "Generic (PLEG): container finished" podID="694f1cf0-fc83-4094-a622-55c80e872163" containerID="138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48" exitCode=0 Oct 30 20:01:42.602651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.602629 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" Oct 30 20:01:42.602651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.602641 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" event={"ID":"694f1cf0-fc83-4094-a622-55c80e872163","Type":"ContainerDied","Data":"138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48"} Oct 30 20:01:42.602999 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.602700 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd" event={"ID":"694f1cf0-fc83-4094-a622-55c80e872163","Type":"ContainerDied","Data":"4e568ad5bc39b1e77640a461cccb7720618850ef0aa20f0a59cc17a43cae4d6d"} Oct 30 20:01:42.602999 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.602725 6514 scope.go:122] "RemoveContainer" containerID="138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48" Oct 30 20:01:42.613113 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.613095 6514 scope.go:122] "RemoveContainer" containerID="138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48" Oct 30 20:01:42.613395 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:01:42.613376 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48\": container with ID starting with 138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48 not found: ID does not exist" containerID="138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48" Oct 30 20:01:42.613442 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.613404 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48"} err="failed to get container status \"138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48\": rpc error: code = NotFound desc = could not find container \"138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48\": container with ID starting with 138b39a411ee74b6ccfb3ba1c6df8b98df4437c1ad3fe08e2131da0cd2ad4b48 not found: ID does not exist" Oct 30 20:01:42.628932 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.628906 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd"] Oct 30 20:01:42.635080 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:42.635051 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cb7041-7bb64cd567-sgrkd"] Oct 30 20:01:43.555167 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:43.555130 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:01:44.159483 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:44.159441 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694f1cf0-fc83-4094-a622-55c80e872163" path="/var/lib/kubelet/pods/694f1cf0-fc83-4094-a622-55c80e872163/volumes" Oct 30 20:01:45.682574 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:45.682536 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerName="payload-processing" containerID="cri-o://e30fb61cae867eff82958f8da649bb699eb6b74794e2b1d352fb1592355ed015" gracePeriod=25 Oct 30 20:01:45.698008 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:45.697975 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerName="payload-pre-processing" containerID="cri-o://5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0" gracePeriod=25 Oct 30 20:01:46.195518 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:46.188584 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:01:46.195518 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:46.188977 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:01:48.872699 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:48.872664 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.188575 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:49.188540 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp"] Oct 30 20:01:49.188823 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:49.188785 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" containerID="cri-o://5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92" gracePeriod=30 Oct 30 20:01:49.814167 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:49.814104 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.321176 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.321142 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" podUID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" containerName="main" containerID="cri-o://59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc" gracePeriod=45 Oct 30 20:01:50.373298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.373270 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" podUID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" containerName="main" probeResult="failure" output="Get \"https://10.134.0.190:8000/ready\": dial tcp 10.134.0.190:8000: connect: connection refused" Oct 30 20:01:50.565681 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.565656 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:01:50.648322 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.648234 6514 generic.go:356] "Generic (PLEG): container finished" podID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" containerID="59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc" exitCode=0 Oct 30 20:01:50.648322 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.648281 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" Oct 30 20:01:50.648545 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.648308 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" event={"ID":"a60b9d9f-1294-4e00-a981-2c51fdc7e808","Type":"ContainerDied","Data":"59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc"} Oct 30 20:01:50.648545 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.648373 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82" event={"ID":"a60b9d9f-1294-4e00-a981-2c51fdc7e808","Type":"ContainerDied","Data":"d0f3b8446c6c6cd3601d2b3260a48564313466e77f4c394fafccbd92e2d30855"} Oct 30 20:01:50.648545 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.648401 6514 scope.go:122] "RemoveContainer" containerID="59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc" Oct 30 20:01:50.660116 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.660097 6514 scope.go:122] "RemoveContainer" containerID="59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc" Oct 30 20:01:50.661117 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:01:50.661094 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc\": container with ID starting with 59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc not found: ID does not exist" containerID="59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc" Oct 30 20:01:50.661266 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.661123 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc"} err="failed to get container status \"59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc\": rpc error: code = NotFound desc = could not find container \"59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc\": container with ID starting with 59e597b2d6da17c528c2d24b69d1acb494fc34adc5eecac98d3e3cc34c0e78bc not found: ID does not exist" Oct 30 20:01:50.662406 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662379 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs\") pod \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " Oct 30 20:01:50.662529 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662445 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir\") pod \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " Oct 30 20:01:50.662529 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662506 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache\") pod \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " Oct 30 20:01:50.662635 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662559 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5\" (UniqueName: \"kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5\") pod \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " Oct 30 20:01:50.662635 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662626 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home\") pod \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " Oct 30 20:01:50.662741 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662727 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm\") pod \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\" (UID: \"a60b9d9f-1294-4e00-a981-2c51fdc7e808\") " Oct 30 20:01:50.662799 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662763 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir" pod "a60b9d9f-1294-4e00-a981-2c51fdc7e808" (UID: "a60b9d9f-1294-4e00-a981-2c51fdc7e808"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.663036 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.662994 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home" pod "a60b9d9f-1294-4e00-a981-2c51fdc7e808" (UID: "a60b9d9f-1294-4e00-a981-2c51fdc7e808"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.663148 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.663046 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache" pod "a60b9d9f-1294-4e00-a981-2c51fdc7e808" (UID: "a60b9d9f-1294-4e00-a981-2c51fdc7e808"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.663211 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.663182 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:50.663211 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.663202 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:50.663305 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.663217 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:50.664684 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.664657 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs" pod "a60b9d9f-1294-4e00-a981-2c51fdc7e808" (UID: "a60b9d9f-1294-4e00-a981-2c51fdc7e808"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:50.664793 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.664771 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5" pod "a60b9d9f-1294-4e00-a981-2c51fdc7e808" (UID: "a60b9d9f-1294-4e00-a981-2c51fdc7e808"). InnerVolumeSpecName "kube-api-access-v4lg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:50.665085 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.665066 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm" pod "a60b9d9f-1294-4e00-a981-2c51fdc7e808" (UID: "a60b9d9f-1294-4e00-a981-2c51fdc7e808"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.770899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.765288 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60b9d9f-1294-4e00-a981-2c51fdc7e808-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:50.770899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.765324 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60b9d9f-1294-4e00-a981-2c51fdc7e808-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:50.770899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.765340 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lg5\" (UniqueName: \"kubernetes.io/projected/a60b9d9f-1294-4e00-a981-2c51fdc7e808-kube-api-access-v4lg5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:50.879755 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:01:50.879715 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:01:50.973345 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.973313 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82"] Oct 30 20:01:50.978327 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:50.978304 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-cb7041/simulator-single-main-cb7041-kserve-7c6677896c-t9g82"] Oct 30 20:01:52.157089 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:52.157053 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" path="/var/lib/kubelet/pods/a60b9d9f-1294-4e00-a981-2c51fdc7e808/volumes" Oct 30 20:01:52.728923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:52.728890 6514 patch_prober.go:29] interesting pod/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:01:52.729094 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:52.728952 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:01:54.439459 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.439436 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:01:54.499192 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499121 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499192 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499166 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499409 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499208 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499409 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499253 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499409 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499286 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499409 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499341 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499409 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499372 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499688 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499466 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499688 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499501 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo\") pod \"53800085-0fac-45e9-a254-7c2a0776eb7d\" (UID: \"53800085-0fac-45e9-a254-7c2a0776eb7d\") " Oct 30 20:01:54.499688 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499538 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:54.499688 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499554 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:54.499944 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499773 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:54.499944 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499929 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.500062 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499949 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.500062 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.499964 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.500176 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.500111 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:54.500314 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.500289 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:54.501774 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.501748 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:54.502106 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.502048 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:54.502106 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.502062 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "kube-api-access-nvsj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:54.502248 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.502144 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token" pod "53800085-0fac-45e9-a254-7c2a0776eb7d" (UID: "53800085-0fac-45e9-a254-7c2a0776eb7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:54.592413 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592380 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx"] Oct 30 20:01:54.592899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592882 6514 cpu_manager.go:414] "containerMap: removing container" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592900 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592920 6514 cpu_manager.go:414] "containerMap: removing container" podUID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" containerName="main" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592925 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" containerName="main" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592940 6514 cpu_manager.go:414] "containerMap: removing container" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592946 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592956 6514 cpu_manager.go:414] "containerMap: removing container" podUID="694f1cf0-fc83-4094-a622-55c80e872163" containerName="maas-api" Oct 30 20:01:54.592959 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.592962 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694f1cf0-fc83-4094-a622-55c80e872163" containerName="maas-api" Oct 30 20:01:54.593178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.593044 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" Oct 30 20:01:54.593178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.593053 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="694f1cf0-fc83-4094-a622-55c80e872163" containerName="maas-api" Oct 30 20:01:54.593178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.593063 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerName="istio-proxy" Oct 30 20:01:54.593178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.593071 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60b9d9f-1294-4e00-a981-2c51fdc7e808" containerName="main" Oct 30 20:01:54.595520 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.595501 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.600650 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.600625 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.600784 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.600652 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvsj9\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-kube-api-access-nvsj9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.600784 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.600663 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.600784 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.600677 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.600784 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.600691 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53800085-0fac-45e9-a254-7c2a0776eb7d-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.600784 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.600703 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53800085-0fac-45e9-a254-7c2a0776eb7d-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:54.615383 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.615360 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx"] Oct 30 20:01:54.670343 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.670295 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.671450 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.671430 6514 generic.go:356] "Generic (PLEG): container finished" podID="53800085-0fac-45e9-a254-7c2a0776eb7d" containerID="5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92" exitCode=0 Oct 30 20:01:54.671541 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.671464 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" event={"ID":"53800085-0fac-45e9-a254-7c2a0776eb7d","Type":"ContainerDied","Data":"5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92"} Oct 30 20:01:54.671541 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.671480 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" Oct 30 20:01:54.671541 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.671517 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp" event={"ID":"53800085-0fac-45e9-a254-7c2a0776eb7d","Type":"ContainerDied","Data":"957ab10e818a381db288824525184cf808a0c33917e8f710ab781d1c37a46f7b"} Oct 30 20:01:54.671668 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.671542 6514 scope.go:122] "RemoveContainer" containerID="5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92" Oct 30 20:01:54.688676 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.688653 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-35dea52b-kserve-self-signed-certs\"" Oct 30 20:01:54.689175 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.689153 6514 scope.go:122] "RemoveContainer" containerID="a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68" Oct 30 20:01:54.697187 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.697172 6514 scope.go:122] "RemoveContainer" containerID="5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92" Oct 30 20:01:54.697451 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:01:54.697432 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92\": container with ID starting with 5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92 not found: ID does not exist" containerID="5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92" Oct 30 20:01:54.697501 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.697461 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92"} err="failed to get container status \"5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92\": rpc error: code = NotFound desc = could not find container \"5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92\": container with ID starting with 5e591dedc2e7fafdd457b245fe73c47e13306e9fdfa9c0fc43442247b80d9e92 not found: ID does not exist" Oct 30 20:01:54.697501 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.697479 6514 scope.go:122] "RemoveContainer" containerID="a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68" Oct 30 20:01:54.697714 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:01:54.697698 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68\": container with ID starting with a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68 not found: ID does not exist" containerID="a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68" Oct 30 20:01:54.697752 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.697719 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68"} err="failed to get container status \"a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68\": rpc error: code = NotFound desc = could not find container \"a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68\": container with ID starting with a78de975c2a8705cdf6b448b7537febd405e2afbe3cfd74cab0b4926c568da68 not found: ID does not exist" Oct 30 20:01:54.700744 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.700718 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp"] Oct 30 20:01:54.701131 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.701111 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.701185 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.701160 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.701252 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.701240 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45b2\" (UniqueName: \"kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.701292 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.701279 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.701402 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.701382 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.701447 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.701416 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.705029 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.705007 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-cb7041-openshift-default-776964d669mzkbp"] Oct 30 20:01:54.802707 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.802617 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45b2\" (UniqueName: \"kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.802707 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.802682 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.802989 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.802748 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.802989 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.802777 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.802989 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.802813 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.802989 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.802891 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.803178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.803153 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.803309 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.803277 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.803432 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.803330 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.805103 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.805083 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.805271 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.805254 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.811524 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.811501 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45b2\" (UniqueName: \"kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2\") pod \"e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:54.906640 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:54.906619 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:55.181962 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:55.181938 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx"] Oct 30 20:01:55.183833 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:01:55.183803 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a7ce3b5_7afd_49d5_8c36_611a072f3b31.slice/crio-1184f9854533fc762907ac48a3255101ba5ee3ccbd218661be2358ccaa927d40 WatchSource:0}: Error finding container 1184f9854533fc762907ac48a3255101ba5ee3ccbd218661be2358ccaa927d40: Status 404 returned error can't find the container with id 1184f9854533fc762907ac48a3255101ba5ee3ccbd218661be2358ccaa927d40 Oct 30 20:01:55.678107 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:55.677781 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" event={"ID":"6a7ce3b5-7afd-49d5-8c36-611a072f3b31","Type":"ContainerStarted","Data":"630aeb0f2bd575002925d9ed4db1d472f9e570cab72873ec3c0e8b759750dc0e"} Oct 30 20:01:55.678107 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:55.677837 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" event={"ID":"6a7ce3b5-7afd-49d5-8c36-611a072f3b31","Type":"ContainerStarted","Data":"1184f9854533fc762907ac48a3255101ba5ee3ccbd218661be2358ccaa927d40"} Oct 30 20:01:55.696257 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:55.696206 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" podStartSLOduration=1.6961934730000001 podStartE2EDuration="1.696193473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:55.695010724 +0000 UTC m=+2685.652829925" watchObservedRunningTime="2026-10-30 20:01:55.696193473 +0000 UTC m=+2685.654012697" Oct 30 20:01:55.907243 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:55.907210 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:01:56.061472 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:56.061438 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:01:56.157588 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:56.157551 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53800085-0fac-45e9-a254-7c2a0776eb7d" path="/var/lib/kubelet/pods/53800085-0fac-45e9-a254-7c2a0776eb7d/volumes" Oct 30 20:01:56.187340 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:56.187302 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:01:56.187340 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:01:56.187333 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:04.906858 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:04.906822 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:02:04.919056 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:04.919026 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:02:05.802263 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:05.802145 6514 generic.go:356] "Generic (PLEG): container finished" podID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerID="e30fb61cae867eff82958f8da649bb699eb6b74794e2b1d352fb1592355ed015" exitCode=0 Oct 30 20:02:05.802263 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:05.802228 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" event={"ID":"ec72bd20-3199-4b14-b963-4ff03634ae0e","Type":"ContainerDied","Data":"e30fb61cae867eff82958f8da649bb699eb6b74794e2b1d352fb1592355ed015"} Oct 30 20:02:05.814607 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:05.814585 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:02:06.008243 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.008215 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:02:06.010539 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.010514 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:02:06.080237 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080209 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume\") pod \"05c501f7-ed94-4a95-8881-a4153f55bb2e\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " Oct 30 20:02:06.080396 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080277 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume\") pod \"ec72bd20-3199-4b14-b963-4ff03634ae0e\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " Oct 30 20:02:06.080396 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080335 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb\" (UniqueName: \"kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb\") pod \"05c501f7-ed94-4a95-8881-a4153f55bb2e\" (UID: \"05c501f7-ed94-4a95-8881-a4153f55bb2e\") " Oct 30 20:02:06.080472 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080434 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq\" (UniqueName: \"kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq\") pod \"ec72bd20-3199-4b14-b963-4ff03634ae0e\" (UID: \"ec72bd20-3199-4b14-b963-4ff03634ae0e\") " Oct 30 20:02:06.080661 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080640 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume" pod "05c501f7-ed94-4a95-8881-a4153f55bb2e" (UID: "05c501f7-ed94-4a95-8881-a4153f55bb2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:06.080716 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080653 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume" pod "ec72bd20-3199-4b14-b963-4ff03634ae0e" (UID: "ec72bd20-3199-4b14-b963-4ff03634ae0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:06.080792 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080772 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c501f7-ed94-4a95-8881-a4153f55bb2e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:06.080792 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.080790 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec72bd20-3199-4b14-b963-4ff03634ae0e-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:06.082493 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.082472 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb" pod "05c501f7-ed94-4a95-8881-a4153f55bb2e" (UID: "05c501f7-ed94-4a95-8881-a4153f55bb2e"). InnerVolumeSpecName "kube-api-access-kclkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:06.082675 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.082654 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq" pod "ec72bd20-3199-4b14-b963-4ff03634ae0e" (UID: "ec72bd20-3199-4b14-b963-4ff03634ae0e"). InnerVolumeSpecName "kube-api-access-xldxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:06.181849 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.181823 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kclkb\" (UniqueName: \"kubernetes.io/projected/05c501f7-ed94-4a95-8881-a4153f55bb2e-kube-api-access-kclkb\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:06.181971 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.181849 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldxq\" (UniqueName: \"kubernetes.io/projected/ec72bd20-3199-4b14-b963-4ff03634ae0e-kube-api-access-xldxq\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:06.808633 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.808602 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" Oct 30 20:02:06.808633 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.808620 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j" event={"ID":"ec72bd20-3199-4b14-b963-4ff03634ae0e","Type":"ContainerDied","Data":"e46c219ecbde3dbc350137e60718fb31f4335013260747b45618df80f28eecee"} Oct 30 20:02:06.809057 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.808691 6514 scope.go:122] "RemoveContainer" containerID="e30fb61cae867eff82958f8da649bb699eb6b74794e2b1d352fb1592355ed015" Oct 30 20:02:06.810164 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.810139 6514 generic.go:356] "Generic (PLEG): container finished" podID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerID="5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0" exitCode=0 Oct 30 20:02:06.810246 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.810191 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" Oct 30 20:02:06.810246 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.810226 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" event={"ID":"05c501f7-ed94-4a95-8881-a4153f55bb2e","Type":"ContainerDied","Data":"5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0"} Oct 30 20:02:06.810353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.810280 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67" event={"ID":"05c501f7-ed94-4a95-8881-a4153f55bb2e","Type":"ContainerDied","Data":"c68eb662693ec3f3e2a3eda18572941e8dc246b8fc744e269ccdc5a25161e494"} Oct 30 20:02:06.817858 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.817844 6514 scope.go:122] "RemoveContainer" containerID="5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0" Oct 30 20:02:06.825724 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.825706 6514 scope.go:122] "RemoveContainer" containerID="5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0" Oct 30 20:02:06.826067 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:02:06.826049 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0\": container with ID starting with 5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0 not found: ID does not exist" containerID="5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0" Oct 30 20:02:06.826136 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.826082 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0"} err="failed to get container status \"5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0\": rpc error: code = NotFound desc = could not find container \"5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0\": container with ID starting with 5fcbb114110abb3ec41ed02447fff421ea6bc36c03a22c7aebda2cb38ed9edc0 not found: ID does not exist" Oct 30 20:02:06.827303 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.827283 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67"] Oct 30 20:02:06.830773 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.830753 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cb7041-68c6d7lt67"] Oct 30 20:02:06.846502 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.846485 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j"] Oct 30 20:02:06.851343 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:06.851325 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cb7041-9c74d8c9dzmn2j"] Oct 30 20:02:08.158092 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:08.158063 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" path="/var/lib/kubelet/pods/05c501f7-ed94-4a95-8881-a4153f55bb2e/volumes" Oct 30 20:02:08.158469 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:08.158436 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" path="/var/lib/kubelet/pods/ec72bd20-3199-4b14-b963-4ff03634ae0e/volumes" Oct 30 20:02:10.204450 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.204411 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77"] Oct 30 20:02:10.205071 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.205052 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerName="payload-processing" Oct 30 20:02:10.205151 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.205074 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerName="payload-processing" Oct 30 20:02:10.205151 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.205122 6514 cpu_manager.go:414] "containerMap: removing container" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerName="payload-pre-processing" Oct 30 20:02:10.205151 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.205131 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerName="payload-pre-processing" Oct 30 20:02:10.205324 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.205232 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec72bd20-3199-4b14-b963-4ff03634ae0e" containerName="payload-processing" Oct 30 20:02:10.205324 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.205249 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c501f7-ed94-4a95-8881-a4153f55bb2e" containerName="payload-pre-processing" Oct 30 20:02:10.208674 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.208652 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.218496 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.218475 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-35dea52b-kserve-self-signed-certs\"" Oct 30 20:02:10.220737 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.220713 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77"] Oct 30 20:02:10.280609 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.280583 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:10.320643 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.320611 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.320835 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.320688 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.320835 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.320748 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchhn\" (UniqueName: \"kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.320835 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.320799 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.321047 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.320889 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.321047 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.320930 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422374 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422338 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422555 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422390 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422555 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422438 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422555 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422463 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchhn\" (UniqueName: \"kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422748 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422595 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422748 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422667 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422900 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422760 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.422900 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422822 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.423012 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.422992 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.424749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.424729 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.424929 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.424914 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.431256 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.431236 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchhn\" (UniqueName: \"kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn\") pod \"e2e-same-id-b-35dea52b-kserve-95dc64889-crp77\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.520989 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.520918 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:10.794812 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.794783 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77"] Oct 30 20:02:10.797406 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:02:10.797377 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3125d8fc_3f0c_4f6a_a935_1b49a441d532.slice/crio-08c5f259a6d1e3782675fac5cd87bd3a6bbcfb5f6e2965d84555c22634d262eb WatchSource:0}: Error finding container 08c5f259a6d1e3782675fac5cd87bd3a6bbcfb5f6e2965d84555c22634d262eb: Status 404 returned error can't find the container with id 08c5f259a6d1e3782675fac5cd87bd3a6bbcfb5f6e2965d84555c22634d262eb Oct 30 20:02:10.819007 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.818984 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 20:02:10.825514 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.825497 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 20:02:10.834313 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:10.834289 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" event={"ID":"3125d8fc-3f0c-4f6a-a935-1b49a441d532","Type":"ContainerStarted","Data":"08c5f259a6d1e3782675fac5cd87bd3a6bbcfb5f6e2965d84555c22634d262eb"} Oct 30 20:02:11.482845 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:11.482807 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.841529 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:11.841446 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" event={"ID":"3125d8fc-3f0c-4f6a-a935-1b49a441d532","Type":"ContainerStarted","Data":"95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21"} Oct 30 20:02:11.860407 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:11.860339 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" podStartSLOduration=1.860322703 podStartE2EDuration="1.860322703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:11.860279762 +0000 UTC m=+2701.818098954" watchObservedRunningTime="2026-10-30 20:02:11.860322703 +0000 UTC m=+2701.818141892" Oct 30 20:02:12.032684 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:12.032649 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:02:12.522075 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:12.522034 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:13.411852 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:13.411804 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.564135 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:13.563985 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.333420 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:19.333383 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.905496 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:19.905463 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.521257 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:20.521223 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:20.533801 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:20.533774 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:20.883712 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:02:20.883619 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:02:20.932970 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:20.932942 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:21.080105 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:21.080064 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.190072 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:21.190044 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.169845 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.169766 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm"] Oct 30 20:02:27.174745 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.174725 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.183189 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.183165 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:02:27.183359 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.183249 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:02:27.183359 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.183297 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:27.183539 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.183370 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 20:02:27.184297 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.184265 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm"] Oct 30 20:02:27.234709 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.234672 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.234842 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.234715 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.235050 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.235022 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxtl\" (UniqueName: \"kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.336035 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.336009 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxtl\" (UniqueName: \"kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.336177 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.336055 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.336177 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.336076 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.336606 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.336588 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.338298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.338280 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.345829 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.345802 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxtl\" (UniqueName: \"kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl\") pod \"maas-api-revoke-sub-452b6d7b-7jfpm\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.490146 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.490111 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:27.734077 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.734040 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:02:27.765426 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.765405 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm"] Oct 30 20:02:27.767420 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:02:27.767388 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6809e174_056f_41a8_8c23_4e278fe5bc9b.slice/crio-29f05629065b6b7ddab722494cf8726c7309b32c732bf4aa5d892d614bc7431e WatchSource:0}: Error finding container 29f05629065b6b7ddab722494cf8726c7309b32c732bf4aa5d892d614bc7431e: Status 404 returned error can't find the container with id 29f05629065b6b7ddab722494cf8726c7309b32c732bf4aa5d892d614bc7431e Oct 30 20:02:27.955772 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:27.955732 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" event={"ID":"6809e174-056f-41a8-8c23-4e278fe5bc9b","Type":"ContainerStarted","Data":"29f05629065b6b7ddab722494cf8726c7309b32c732bf4aa5d892d614bc7431e"} Oct 30 20:02:28.654042 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:28.654004 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.965690 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:28.965652 6514 generic.go:356] "Generic (PLEG): container finished" podID="6809e174-056f-41a8-8c23-4e278fe5bc9b" containerID="b431fb8bc7dc32b780f7672732d1832ae7f1debedc1859ca75280c94a616bee2" exitCode=0 Oct 30 20:02:28.965945 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:28.965749 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" event={"ID":"6809e174-056f-41a8-8c23-4e278fe5bc9b","Type":"ContainerDied","Data":"b431fb8bc7dc32b780f7672732d1832ae7f1debedc1859ca75280c94a616bee2"} Oct 30 20:02:30.644928 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.644064 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:30.826899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.814168 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl\" (UniqueName: \"kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl\") pod \"6809e174-056f-41a8-8c23-4e278fe5bc9b\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " Oct 30 20:02:30.826899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.814337 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca\") pod \"6809e174-056f-41a8-8c23-4e278fe5bc9b\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " Oct 30 20:02:30.826899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.814381 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca\") pod \"6809e174-056f-41a8-8c23-4e278fe5bc9b\" (UID: \"6809e174-056f-41a8-8c23-4e278fe5bc9b\") " Oct 30 20:02:30.859893 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.855000 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca" pod "6809e174-056f-41a8-8c23-4e278fe5bc9b" (UID: "6809e174-056f-41a8-8c23-4e278fe5bc9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:30.923940 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.921047 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:30.923940 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.921749 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl" pod "6809e174-056f-41a8-8c23-4e278fe5bc9b" (UID: "6809e174-056f-41a8-8c23-4e278fe5bc9b"). InnerVolumeSpecName "kube-api-access-2wxtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:30.998888 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:30.987378 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca" pod "6809e174-056f-41a8-8c23-4e278fe5bc9b" (UID: "6809e174-056f-41a8-8c23-4e278fe5bc9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:31.049910 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:31.036705 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxtl\" (UniqueName: \"kubernetes.io/projected/6809e174-056f-41a8-8c23-4e278fe5bc9b-kube-api-access-2wxtl\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:31.049910 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:31.036750 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6809e174-056f-41a8-8c23-4e278fe5bc9b-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:31.064908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:31.061938 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" event={"ID":"6809e174-056f-41a8-8c23-4e278fe5bc9b","Type":"ContainerDied","Data":"29f05629065b6b7ddab722494cf8726c7309b32c732bf4aa5d892d614bc7431e"} Oct 30 20:02:31.064908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:31.062005 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f05629065b6b7ddab722494cf8726c7309b32c732bf4aa5d892d614bc7431e" Oct 30 20:02:31.064908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:31.062081 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm" Oct 30 20:02:38.124005 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:38.123966 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx"] Oct 30 20:02:38.200333 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:38.200295 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.622811 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:39.622778 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:02:40.239592 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:40.239556 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77"] Oct 30 20:02:40.301661 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:40.301614 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:42.336887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:42.330015 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qbqj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:02:42.336887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:42.330095 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qbqj6" podUID="91f1db0f-3030-450b-8a45-83b100e58823" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:43.943064 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.943018 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps"] Oct 30 20:02:43.943584 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.943490 6514 cpu_manager.go:414] "containerMap: removing container" podUID="6809e174-056f-41a8-8c23-4e278fe5bc9b" containerName="revoke-keys" Oct 30 20:02:43.943584 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.943510 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6809e174-056f-41a8-8c23-4e278fe5bc9b" containerName="revoke-keys" Oct 30 20:02:43.943737 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.943638 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="6809e174-056f-41a8-8c23-4e278fe5bc9b" containerName="revoke-keys" Oct 30 20:02:43.946746 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.946723 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:43.950900 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.950850 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:43.952271 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.952244 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:02:43.952271 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.952244 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 20:02:43.952454 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.952348 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:02:43.959234 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:43.959211 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps"] Oct 30 20:02:44.117397 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.117356 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmrc\" (UniqueName: \"kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.117611 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.117505 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.117611 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.117575 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.218677 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.218590 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.218677 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.218662 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.218961 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.218877 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmrc\" (UniqueName: \"kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.219371 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.219350 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.221812 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.221785 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.229603 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.229577 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmrc\" (UniqueName: \"kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc\") pod \"maas-api-revoke-sub-3e31a506-q6qps\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.262898 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.262876 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:44.320721 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.319915 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:02:44.550772 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:44.550745 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps"] Oct 30 20:02:44.552774 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:02:44.552739 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ac5339_2d81_4ea2_a8ce_3e61910fa061.slice/crio-84abdd08ff5f690a01f58312e43081917c5dd6e10aff9b8d18b439f6ea541a52 WatchSource:0}: Error finding container 84abdd08ff5f690a01f58312e43081917c5dd6e10aff9b8d18b439f6ea541a52: Status 404 returned error can't find the container with id 84abdd08ff5f690a01f58312e43081917c5dd6e10aff9b8d18b439f6ea541a52 Oct 30 20:02:45.169667 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:45.169561 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" event={"ID":"17ac5339-2d81-4ea2-a8ce-3e61910fa061","Type":"ContainerStarted","Data":"448d1bb0fe6fb9ebf760feb8b1934dad9de35d9a5377fc5296203fccae5b038a"} Oct 30 20:02:45.169667 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:45.169628 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" event={"ID":"17ac5339-2d81-4ea2-a8ce-3e61910fa061","Type":"ContainerStarted","Data":"84abdd08ff5f690a01f58312e43081917c5dd6e10aff9b8d18b439f6ea541a52"} Oct 30 20:02:45.193492 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:45.193417 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" podStartSLOduration=2.193397757 podStartE2EDuration="2.193397757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:45.189930782 +0000 UTC m=+2735.147749971" watchObservedRunningTime="2026-10-30 20:02:45.193397757 +0000 UTC m=+2735.151216946" Oct 30 20:02:46.188887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:46.187242 6514 generic.go:356] "Generic (PLEG): container finished" podID="17ac5339-2d81-4ea2-a8ce-3e61910fa061" containerID="448d1bb0fe6fb9ebf760feb8b1934dad9de35d9a5377fc5296203fccae5b038a" exitCode=0 Oct 30 20:02:46.195886 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:46.193319 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" event={"ID":"17ac5339-2d81-4ea2-a8ce-3e61910fa061","Type":"ContainerDied","Data":"448d1bb0fe6fb9ebf760feb8b1934dad9de35d9a5377fc5296203fccae5b038a"} Oct 30 20:02:47.579954 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.579922 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:47.705197 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.705165 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc\" (UniqueName: \"kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc\") pod \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " Oct 30 20:02:47.705402 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.705225 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca\") pod \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " Oct 30 20:02:47.705402 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.705283 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca\") pod \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\" (UID: \"17ac5339-2d81-4ea2-a8ce-3e61910fa061\") " Oct 30 20:02:47.705743 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.705717 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca" pod "17ac5339-2d81-4ea2-a8ce-3e61910fa061" (UID: "17ac5339-2d81-4ea2-a8ce-3e61910fa061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:47.707313 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.707290 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc" pod "17ac5339-2d81-4ea2-a8ce-3e61910fa061" (UID: "17ac5339-2d81-4ea2-a8ce-3e61910fa061"). InnerVolumeSpecName "kube-api-access-9lmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:47.707396 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.707319 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca" pod "17ac5339-2d81-4ea2-a8ce-3e61910fa061" (UID: "17ac5339-2d81-4ea2-a8ce-3e61910fa061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:47.806608 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.806573 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lmrc\" (UniqueName: \"kubernetes.io/projected/17ac5339-2d81-4ea2-a8ce-3e61910fa061-kube-api-access-9lmrc\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:47.806608 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.806606 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:47.806838 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:47.806621 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ac5339-2d81-4ea2-a8ce-3e61910fa061-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:48.292328 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:48.292303 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" Oct 30 20:02:48.292487 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:48.292304 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps" event={"ID":"17ac5339-2d81-4ea2-a8ce-3e61910fa061","Type":"ContainerDied","Data":"84abdd08ff5f690a01f58312e43081917c5dd6e10aff9b8d18b439f6ea541a52"} Oct 30 20:02:48.292487 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:48.292418 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84abdd08ff5f690a01f58312e43081917c5dd6e10aff9b8d18b439f6ea541a52" Oct 30 20:02:50.879644 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:02:50.879604 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:02:53.144924 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.144849 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" podUID="6a7ce3b5-7afd-49d5-8c36-611a072f3b31" containerName="main" containerID="cri-o://630aeb0f2bd575002925d9ed4db1d472f9e570cab72873ec3c0e8b759750dc0e" gracePeriod=45 Oct 30 20:02:53.320254 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.320217 6514 generic.go:356] "Generic (PLEG): container finished" podID="6a7ce3b5-7afd-49d5-8c36-611a072f3b31" containerID="630aeb0f2bd575002925d9ed4db1d472f9e570cab72873ec3c0e8b759750dc0e" exitCode=0 Oct 30 20:02:53.320384 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.320282 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" event={"ID":"6a7ce3b5-7afd-49d5-8c36-611a072f3b31","Type":"ContainerDied","Data":"630aeb0f2bd575002925d9ed4db1d472f9e570cab72873ec3c0e8b759750dc0e"} Oct 30 20:02:53.401327 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.401269 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:02:53.560126 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560079 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache\") pod \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " Oct 30 20:02:53.560330 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560145 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir\") pod \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " Oct 30 20:02:53.560330 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560195 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2\" (UniqueName: \"kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2\") pod \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " Oct 30 20:02:53.560330 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560233 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs\") pod \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " Oct 30 20:02:53.560330 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560252 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm\") pod \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " Oct 30 20:02:53.560330 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560315 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home\") pod \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\" (UID: \"6a7ce3b5-7afd-49d5-8c36-611a072f3b31\") " Oct 30 20:02:53.560574 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560365 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache" pod "6a7ce3b5-7afd-49d5-8c36-611a072f3b31" (UID: "6a7ce3b5-7afd-49d5-8c36-611a072f3b31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.560574 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560385 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir" pod "6a7ce3b5-7afd-49d5-8c36-611a072f3b31" (UID: "6a7ce3b5-7afd-49d5-8c36-611a072f3b31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.560695 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560659 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home" pod "6a7ce3b5-7afd-49d5-8c36-611a072f3b31" (UID: "6a7ce3b5-7afd-49d5-8c36-611a072f3b31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.560732 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560690 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.560732 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.560706 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.562482 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.562458 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs" pod "6a7ce3b5-7afd-49d5-8c36-611a072f3b31" (UID: "6a7ce3b5-7afd-49d5-8c36-611a072f3b31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:53.562764 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.562743 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm" pod "6a7ce3b5-7afd-49d5-8c36-611a072f3b31" (UID: "6a7ce3b5-7afd-49d5-8c36-611a072f3b31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.562812 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.562777 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2" pod "6a7ce3b5-7afd-49d5-8c36-611a072f3b31" (UID: "6a7ce3b5-7afd-49d5-8c36-611a072f3b31"). InnerVolumeSpecName "kube-api-access-x45b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:53.662027 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.661948 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x45b2\" (UniqueName: \"kubernetes.io/projected/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-kube-api-access-x45b2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.662027 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.661979 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.662027 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.661993 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.662027 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:53.662004 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7ce3b5-7afd-49d5-8c36-611a072f3b31-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.327617 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:54.327590 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" Oct 30 20:02:54.328074 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:54.327626 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx" event={"ID":"6a7ce3b5-7afd-49d5-8c36-611a072f3b31","Type":"ContainerDied","Data":"1184f9854533fc762907ac48a3255101ba5ee3ccbd218661be2358ccaa927d40"} Oct 30 20:02:54.328074 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:54.327683 6514 scope.go:122] "RemoveContainer" containerID="630aeb0f2bd575002925d9ed4db1d472f9e570cab72873ec3c0e8b759750dc0e" Oct 30 20:02:54.347603 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:54.347579 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx"] Oct 30 20:02:54.351923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:54.351900 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-35dea52b-kserve-659bf46b8f-f2jgx"] Oct 30 20:02:54.760838 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:54.760802 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:02:55.277939 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.277859 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" podUID="3125d8fc-3f0c-4f6a-a935-1b49a441d532" containerName="main" containerID="cri-o://95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21" gracePeriod=45 Oct 30 20:02:55.527507 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.527481 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:55.682595 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682570 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache\") pod \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " Oct 30 20:02:55.682740 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682659 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn\" (UniqueName: \"kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn\") pod \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " Oct 30 20:02:55.682740 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682722 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home\") pod \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " Oct 30 20:02:55.682821 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682749 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir\") pod \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " Oct 30 20:02:55.682890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682828 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs\" (UniqueName: \"kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs\") pod \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " Oct 30 20:02:55.682890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682854 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache" pod "3125d8fc-3f0c-4f6a-a935-1b49a441d532" (UID: "3125d8fc-3f0c-4f6a-a935-1b49a441d532"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:55.682983 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.682909 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm\") pod \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\" (UID: \"3125d8fc-3f0c-4f6a-a935-1b49a441d532\") " Oct 30 20:02:55.683088 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.683028 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home" pod "3125d8fc-3f0c-4f6a-a935-1b49a441d532" (UID: "3125d8fc-3f0c-4f6a-a935-1b49a441d532"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:55.683088 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.683083 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir" pod "3125d8fc-3f0c-4f6a-a935-1b49a441d532" (UID: "3125d8fc-3f0c-4f6a-a935-1b49a441d532"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:55.683306 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.683288 6514 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-home\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:55.683362 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.683312 6514 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tmp-dir\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:55.683362 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.683327 6514 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-model-cache\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:55.684706 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.684676 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn" pod "3125d8fc-3f0c-4f6a-a935-1b49a441d532" (UID: "3125d8fc-3f0c-4f6a-a935-1b49a441d532"). InnerVolumeSpecName "kube-api-access-xchhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:55.684706 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.684694 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs" pod "3125d8fc-3f0c-4f6a-a935-1b49a441d532" (UID: "3125d8fc-3f0c-4f6a-a935-1b49a441d532"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:55.684844 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.684809 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm" pod "3125d8fc-3f0c-4f6a-a935-1b49a441d532" (UID: "3125d8fc-3f0c-4f6a-a935-1b49a441d532"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:55.784786 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.784761 6514 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3125d8fc-3f0c-4f6a-a935-1b49a441d532-tls-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:55.784786 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.784783 6514 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3125d8fc-3f0c-4f6a-a935-1b49a441d532-dshm\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:55.784991 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:55.784793 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xchhn\" (UniqueName: \"kubernetes.io/projected/3125d8fc-3f0c-4f6a-a935-1b49a441d532-kube-api-access-xchhn\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:56.157065 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.156990 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7ce3b5-7afd-49d5-8c36-611a072f3b31" path="/var/lib/kubelet/pods/6a7ce3b5-7afd-49d5-8c36-611a072f3b31/volumes" Oct 30 20:02:56.341254 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.341218 6514 generic.go:356] "Generic (PLEG): container finished" podID="3125d8fc-3f0c-4f6a-a935-1b49a441d532" containerID="95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21" exitCode=0 Oct 30 20:02:56.341254 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.341264 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" Oct 30 20:02:56.341469 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.341278 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" event={"ID":"3125d8fc-3f0c-4f6a-a935-1b49a441d532","Type":"ContainerDied","Data":"95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21"} Oct 30 20:02:56.341469 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.341319 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77" event={"ID":"3125d8fc-3f0c-4f6a-a935-1b49a441d532","Type":"ContainerDied","Data":"08c5f259a6d1e3782675fac5cd87bd3a6bbcfb5f6e2965d84555c22634d262eb"} Oct 30 20:02:56.341469 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.341336 6514 scope.go:122] "RemoveContainer" containerID="95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21" Oct 30 20:02:56.351677 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.351660 6514 scope.go:122] "RemoveContainer" containerID="95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21" Oct 30 20:02:56.351975 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:02:56.351955 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21\": container with ID starting with 95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21 not found: ID does not exist" containerID="95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21" Oct 30 20:02:56.352041 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.351988 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21"} err="failed to get container status \"95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21\": rpc error: code = NotFound desc = could not find container \"95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21\": container with ID starting with 95c71897b74cff1389a0ad67b9abd5a17350d0d496c90a4594fc8c0ba037fb21 not found: ID does not exist" Oct 30 20:02:56.366186 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.366162 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77"] Oct 30 20:02:56.369189 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:56.369169 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-35dea52b-kserve-95dc64889-crp77"] Oct 30 20:02:58.367889 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:58.349110 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3125d8fc-3f0c-4f6a-a935-1b49a441d532" path="/var/lib/kubelet/pods/3125d8fc-3f0c-4f6a-a935-1b49a441d532/volumes" Oct 30 20:02:58.702887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:02:58.700171 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:03:02.429653 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.429620 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx"] Oct 30 20:03:02.430067 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430035 6514 cpu_manager.go:414] "containerMap: removing container" podUID="17ac5339-2d81-4ea2-a8ce-3e61910fa061" containerName="revoke-keys" Oct 30 20:03:02.430067 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430051 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ac5339-2d81-4ea2-a8ce-3e61910fa061" containerName="revoke-keys" Oct 30 20:03:02.430157 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430072 6514 cpu_manager.go:414] "containerMap: removing container" podUID="3125d8fc-3f0c-4f6a-a935-1b49a441d532" containerName="main" Oct 30 20:03:02.430157 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430078 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3125d8fc-3f0c-4f6a-a935-1b49a441d532" containerName="main" Oct 30 20:03:02.430157 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430089 6514 cpu_manager.go:414] "containerMap: removing container" podUID="6a7ce3b5-7afd-49d5-8c36-611a072f3b31" containerName="main" Oct 30 20:03:02.430157 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430094 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7ce3b5-7afd-49d5-8c36-611a072f3b31" containerName="main" Oct 30 20:03:02.430307 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430162 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ac5339-2d81-4ea2-a8ce-3e61910fa061" containerName="revoke-keys" Oct 30 20:03:02.430307 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430174 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="3125d8fc-3f0c-4f6a-a935-1b49a441d532" containerName="main" Oct 30 20:03:02.430307 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.430181 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7ce3b5-7afd-49d5-8c36-611a072f3b31" containerName="main" Oct 30 20:03:02.434805 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.434779 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.446070 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.444850 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:02.446243 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.445533 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:03:02.446243 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.445595 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:03:02.446243 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.446021 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 20:03:02.446404 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.446257 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx"] Oct 30 20:03:02.587742 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.587711 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckr5\" (UniqueName: \"kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.587950 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.587773 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.587950 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.587897 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.689509 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.689442 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.689638 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.689516 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckr5\" (UniqueName: \"kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.689638 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.689601 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.690253 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.690232 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.691828 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.691809 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.699218 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.699196 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckr5\" (UniqueName: \"kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5\") pod \"maas-api-revoke-sub-2aa221a4-c6chx\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:02.749030 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:02.749009 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:03.021743 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:03.021694 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx"] Oct 30 20:03:03.024145 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:03.024120 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecb9a78_35da_4298_921e_29fc4ce51888.slice/crio-e8b835b20fb1067dd04c8fa690bfa5a93376cf090e3c03c830467f9df5b5b9b2 WatchSource:0}: Error finding container e8b835b20fb1067dd04c8fa690bfa5a93376cf090e3c03c830467f9df5b5b9b2: Status 404 returned error can't find the container with id e8b835b20fb1067dd04c8fa690bfa5a93376cf090e3c03c830467f9df5b5b9b2 Oct 30 20:03:03.393499 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:03.393463 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" event={"ID":"1ecb9a78-35da-4298-921e-29fc4ce51888","Type":"ContainerStarted","Data":"5b7d50196b9906210e470bf1b1d883f99872cdae848b9088477bb25fe7c638fd"} Oct 30 20:03:03.393630 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:03.393515 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" event={"ID":"1ecb9a78-35da-4298-921e-29fc4ce51888","Type":"ContainerStarted","Data":"e8b835b20fb1067dd04c8fa690bfa5a93376cf090e3c03c830467f9df5b5b9b2"} Oct 30 20:03:03.417010 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:03.416970 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" podStartSLOduration=1.416955587 podStartE2EDuration="1.416955587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:03.414032043 +0000 UTC m=+2753.371851232" watchObservedRunningTime="2026-10-30 20:03:03.416955587 +0000 UTC m=+2753.374774772" Oct 30 20:03:04.400124 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:04.400094 6514 generic.go:356] "Generic (PLEG): container finished" podID="1ecb9a78-35da-4298-921e-29fc4ce51888" containerID="5b7d50196b9906210e470bf1b1d883f99872cdae848b9088477bb25fe7c638fd" exitCode=0 Oct 30 20:03:04.400525 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:04.400181 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" event={"ID":"1ecb9a78-35da-4298-921e-29fc4ce51888","Type":"ContainerDied","Data":"5b7d50196b9906210e470bf1b1d883f99872cdae848b9088477bb25fe7c638fd"} Oct 30 20:03:05.534144 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.534115 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:05.612602 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.612568 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5\" (UniqueName: \"kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5\") pod \"1ecb9a78-35da-4298-921e-29fc4ce51888\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " Oct 30 20:03:05.612747 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.612614 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca\") pod \"1ecb9a78-35da-4298-921e-29fc4ce51888\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " Oct 30 20:03:05.612747 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.612733 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca\") pod \"1ecb9a78-35da-4298-921e-29fc4ce51888\" (UID: \"1ecb9a78-35da-4298-921e-29fc4ce51888\") " Oct 30 20:03:05.613130 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.613099 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca" pod "1ecb9a78-35da-4298-921e-29fc4ce51888" (UID: "1ecb9a78-35da-4298-921e-29fc4ce51888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:05.614713 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.614688 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca" pod "1ecb9a78-35da-4298-921e-29fc4ce51888" (UID: "1ecb9a78-35da-4298-921e-29fc4ce51888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:05.614849 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.614830 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5" pod "1ecb9a78-35da-4298-921e-29fc4ce51888" (UID: "1ecb9a78-35da-4298-921e-29fc4ce51888"). InnerVolumeSpecName "kube-api-access-2ckr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:05.713837 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.713810 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-xks-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:05.713837 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.713835 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckr5\" (UniqueName: \"kubernetes.io/projected/1ecb9a78-35da-4298-921e-29fc4ce51888-kube-api-access-2ckr5\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:05.714016 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:05.713845 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb9a78-35da-4298-921e-29fc4ce51888-maas-api-openshift-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:06.411603 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:06.411566 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" event={"ID":"1ecb9a78-35da-4298-921e-29fc4ce51888","Type":"ContainerDied","Data":"e8b835b20fb1067dd04c8fa690bfa5a93376cf090e3c03c830467f9df5b5b9b2"} Oct 30 20:03:06.411603 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:06.411604 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx" Oct 30 20:03:06.411804 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:06.411620 6514 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b835b20fb1067dd04c8fa690bfa5a93376cf090e3c03c830467f9df5b5b9b2" Oct 30 20:03:11.022878 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:11.022805 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6"] Oct 30 20:03:11.025155 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:11.025130 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wh2b6"] Oct 30 20:03:12.157420 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:12.157376 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540a3e30-6c89-4ca5-ad58-b870efa3c87e" path="/var/lib/kubelet/pods/540a3e30-6c89-4ca5-ad58-b870efa3c87e/volumes" Oct 30 20:03:13.261578 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:13.261542 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:03:13.336662 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:13.336631 6514 scope.go:122] "RemoveContainer" containerID="2d0bf3d688e4649d58ec508be48a49af904b34084bc3a13a8e7df8a5fdbcea1c" Oct 30 20:03:13.612187 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:13.612062 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.662267 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.662233 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh"] Oct 30 20:03:14.662658 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.662649 6514 cpu_manager.go:414] "containerMap: removing container" podUID="1ecb9a78-35da-4298-921e-29fc4ce51888" containerName="revoke-keys" Oct 30 20:03:14.662706 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.662660 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecb9a78-35da-4298-921e-29fc4ce51888" containerName="revoke-keys" Oct 30 20:03:14.662749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.662740 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecb9a78-35da-4298-921e-29fc4ce51888" containerName="revoke-keys" Oct 30 20:03:14.667468 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.667440 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.676898 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.675432 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh"] Oct 30 20:03:14.678913 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.678887 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:03:14.679019 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.679010 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:03:14.679110 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.679095 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:03:14.679208 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.679194 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2z7lq\"" Oct 30 20:03:14.679251 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.679241 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:03:14.685234 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.685214 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76"] Oct 30 20:03:14.688584 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.688566 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.702356 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.702335 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76"] Oct 30 20:03:14.704923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.703177 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.704923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.703335 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.704923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.703387 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.704923 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.703439 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrjt\" (UniqueName: \"kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.804292 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804256 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.804292 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804293 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.804555 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804329 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6722016-06cb-4680-a1fa-8737d685e461-openshift-service-ca\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.804555 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804354 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6722016-06cb-4680-a1fa-8737d685e461-maas-api-tls\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.804555 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804506 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.804729 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804566 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7299\" (UniqueName: \"kubernetes.io/projected/d6722016-06cb-4680-a1fa-8737d685e461-kube-api-access-f7299\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.804729 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804611 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrjt\" (UniqueName: \"kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.804856 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.804791 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6722016-06cb-4680-a1fa-8737d685e461-metrics-certs\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.805225 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.805201 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.806953 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.806928 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.807099 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.807004 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.813595 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.813571 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrjt\" (UniqueName: \"kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt\") pod \"maas-api-585bcbc4bb-sd9mh\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:14.906109 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.906068 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6722016-06cb-4680-a1fa-8737d685e461-metrics-certs\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.906297 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.906138 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6722016-06cb-4680-a1fa-8737d685e461-openshift-service-ca\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.906297 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.906160 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6722016-06cb-4680-a1fa-8737d685e461-maas-api-tls\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.906297 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.906197 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7299\" (UniqueName: \"kubernetes.io/projected/d6722016-06cb-4680-a1fa-8737d685e461-kube-api-access-f7299\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.906973 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.906943 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6722016-06cb-4680-a1fa-8737d685e461-openshift-service-ca\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.908677 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.908657 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6722016-06cb-4680-a1fa-8737d685e461-maas-api-tls\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.908977 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.908955 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6722016-06cb-4680-a1fa-8737d685e461-metrics-certs\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.914263 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.914206 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7299\" (UniqueName: \"kubernetes.io/projected/d6722016-06cb-4680-a1fa-8737d685e461-kube-api-access-f7299\") pod \"maas-api-79c7b97c4b-9xh76\" (UID: \"d6722016-06cb-4680-a1fa-8737d685e461\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:14.987328 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:14.987292 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:15.009519 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:15.009487 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:15.269596 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:15.269566 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh"] Oct 30 20:03:15.271399 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:15.271365 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ae7c1e_5173_4ed5_828a_8627fc7e2920.slice/crio-30e3dce81ae5ace922b08cfc969b5aad4092ab8f6795309ea7e2d9efa7fa5bb0 WatchSource:0}: Error finding container 30e3dce81ae5ace922b08cfc969b5aad4092ab8f6795309ea7e2d9efa7fa5bb0: Status 404 returned error can't find the container with id 30e3dce81ae5ace922b08cfc969b5aad4092ab8f6795309ea7e2d9efa7fa5bb0 Oct 30 20:03:15.272674 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:15.272656 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:03:15.298198 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:15.298171 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76"] Oct 30 20:03:15.302420 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:15.302385 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6722016_06cb_4680_a1fa_8737d685e461.slice/crio-d3cbeacceaa32baa18b07edcbde56a56ea5fb86e1a6376731700cdf3b7c62a4e WatchSource:0}: Error finding container d3cbeacceaa32baa18b07edcbde56a56ea5fb86e1a6376731700cdf3b7c62a4e: Status 404 returned error can't find the container with id d3cbeacceaa32baa18b07edcbde56a56ea5fb86e1a6376731700cdf3b7c62a4e Oct 30 20:03:15.466254 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:15.466219 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" event={"ID":"b6ae7c1e-5173-4ed5-828a-8627fc7e2920","Type":"ContainerStarted","Data":"30e3dce81ae5ace922b08cfc969b5aad4092ab8f6795309ea7e2d9efa7fa5bb0"} Oct 30 20:03:15.467603 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:15.467570 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" event={"ID":"d6722016-06cb-4680-a1fa-8737d685e461","Type":"ContainerStarted","Data":"d3cbeacceaa32baa18b07edcbde56a56ea5fb86e1a6376731700cdf3b7c62a4e"} Oct 30 20:03:16.473804 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:16.473760 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" event={"ID":"d6722016-06cb-4680-a1fa-8737d685e461","Type":"ContainerStarted","Data":"783f85790a8e398d72f96a8bdad8532ce3e4162d6d9d45d0a02e04c453581f3d"} Oct 30 20:03:16.474221 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:16.473880 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:16.475366 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:16.475339 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" event={"ID":"b6ae7c1e-5173-4ed5-828a-8627fc7e2920","Type":"ContainerStarted","Data":"eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be"} Oct 30 20:03:16.475462 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:16.475429 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:16.495997 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:16.495957 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" podStartSLOduration=1.9902605690000001 podStartE2EDuration="2.495945742s" totalImagesPullingTime="505.685173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:16.493327732 +0000 UTC m=+2766.451146922" watchObservedRunningTime="2026-10-30 20:03:16.495945742 +0000 UTC m=+2766.453764931" Oct 30 20:03:16.514230 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:16.514194 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" podStartSLOduration=1.999438914 podStartE2EDuration="2.514181424s" totalImagesPullingTime="514.74251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:16.51268101 +0000 UTC m=+2766.470500200" watchObservedRunningTime="2026-10-30 20:03:16.514181424 +0000 UTC m=+2766.472000613" Oct 30 20:03:20.882001 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:03:20.881948 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:03:21.678227 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:21.678192 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.054406 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.054371 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26"] Oct 30 20:03:22.058721 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.058699 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.069201 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.069171 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-58847295-openshift-default-dockercfg-t9w8k\"" Oct 30 20:03:22.071228 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.071207 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26"] Oct 30 20:03:22.135994 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.135963 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.182379 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182347 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182546 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182381 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182546 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182410 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182546 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182482 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182686 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182558 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182686 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182588 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvf9\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182686 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182654 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182686 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182675 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.182909 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.182744 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.260426 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.260342 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.283534 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283498 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.283534 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283534 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvf9\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.283770 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283564 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.283770 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283677 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.283770 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283726 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284005 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283846 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284005 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283894 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284005 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283946 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284005 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.283979 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284220 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.284091 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284220 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.284133 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284346 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.284287 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284405 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.284371 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.284491 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.284465 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.285943 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.285921 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.286468 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.286449 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.293817 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.293784 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.293927 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.293903 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvf9\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9\") pod \"e2e-ait-58847295-openshift-default-8f844cdfc-m4s26\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.374579 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.374524 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:22.486192 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.486158 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-9xh76" Oct 30 20:03:22.486611 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.486590 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:22.567753 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.567720 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh"] Oct 30 20:03:22.567986 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.567956 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" podUID="b6ae7c1e-5173-4ed5-828a-8627fc7e2920" containerName="maas-api" containerID="cri-o://eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be" gracePeriod=30 Oct 30 20:03:22.657314 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.657284 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26"] Oct 30 20:03:22.659518 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:22.659494 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc77c52_2cc6_4282_bde2_3cb45085f959.slice/crio-c658b965305e57cbad03e230f7731d7478587dce09ef3f5a7438c4c3f95fd162 WatchSource:0}: Error finding container c658b965305e57cbad03e230f7731d7478587dce09ef3f5a7438c4c3f95fd162: Status 404 returned error can't find the container with id c658b965305e57cbad03e230f7731d7478587dce09ef3f5a7438c4c3f95fd162 Oct 30 20:03:22.673755 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.673724 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:22.673821 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.673794 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:22.673857 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.673824 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:22.811260 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.811235 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:22.890131 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.890101 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs\") pod \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " Oct 30 20:03:22.890319 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.890149 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca\") pod \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " Oct 30 20:03:22.890319 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.890257 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt\" (UniqueName: \"kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt\") pod \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " Oct 30 20:03:22.890436 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.890341 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls\") pod \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\" (UID: \"b6ae7c1e-5173-4ed5-828a-8627fc7e2920\") " Oct 30 20:03:22.890597 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.890570 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca" pod "b6ae7c1e-5173-4ed5-828a-8627fc7e2920" (UID: "b6ae7c1e-5173-4ed5-828a-8627fc7e2920"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:22.890791 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.890760 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:22.892481 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.892458 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls" pod "b6ae7c1e-5173-4ed5-828a-8627fc7e2920" (UID: "b6ae7c1e-5173-4ed5-828a-8627fc7e2920"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:22.892712 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.892689 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs" pod "b6ae7c1e-5173-4ed5-828a-8627fc7e2920" (UID: "b6ae7c1e-5173-4ed5-828a-8627fc7e2920"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:22.892778 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.892749 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt" pod "b6ae7c1e-5173-4ed5-828a-8627fc7e2920" (UID: "b6ae7c1e-5173-4ed5-828a-8627fc7e2920"). InnerVolumeSpecName "kube-api-access-qdrjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:22.992311 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.992282 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:22.992311 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.992308 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:22.992514 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:22.992320 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdrjt\" (UniqueName: \"kubernetes.io/projected/b6ae7c1e-5173-4ed5-828a-8627fc7e2920-kube-api-access-qdrjt\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:23.509564 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.509528 6514 generic.go:356] "Generic (PLEG): container finished" podID="b6ae7c1e-5173-4ed5-828a-8627fc7e2920" containerID="eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be" exitCode=0 Oct 30 20:03:23.510062 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.509581 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" Oct 30 20:03:23.510062 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.509610 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" event={"ID":"b6ae7c1e-5173-4ed5-828a-8627fc7e2920","Type":"ContainerDied","Data":"eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be"} Oct 30 20:03:23.510062 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.509674 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh" event={"ID":"b6ae7c1e-5173-4ed5-828a-8627fc7e2920","Type":"ContainerDied","Data":"30e3dce81ae5ace922b08cfc969b5aad4092ab8f6795309ea7e2d9efa7fa5bb0"} Oct 30 20:03:23.510062 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.509706 6514 scope.go:122] "RemoveContainer" containerID="eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be" Oct 30 20:03:23.511530 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.511501 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" event={"ID":"9fc77c52-2cc6-4282-bde2-3cb45085f959","Type":"ContainerStarted","Data":"71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4"} Oct 30 20:03:23.511638 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.511549 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" event={"ID":"9fc77c52-2cc6-4282-bde2-3cb45085f959","Type":"ContainerStarted","Data":"c658b965305e57cbad03e230f7731d7478587dce09ef3f5a7438c4c3f95fd162"} Oct 30 20:03:23.519511 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.519492 6514 scope.go:122] "RemoveContainer" containerID="eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be" Oct 30 20:03:23.519759 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:03:23.519741 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be\": container with ID starting with eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be not found: ID does not exist" containerID="eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be" Oct 30 20:03:23.519813 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.519769 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be"} err="failed to get container status \"eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be\": rpc error: code = NotFound desc = could not find container \"eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be\": container with ID starting with eedaf6c1bff83beb149dfa02f844c001d4ffaabc980126dbac90e4c1eb9e11be not found: ID does not exist" Oct 30 20:03:23.539730 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.539687 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podStartSLOduration=1.539671888 podStartE2EDuration="1.539671888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:23.538982905 +0000 UTC m=+2773.496802534" watchObservedRunningTime="2026-10-30 20:03:23.539671888 +0000 UTC m=+2773.497491079" Oct 30 20:03:23.557028 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.556962 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh"] Oct 30 20:03:23.560727 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:23.560704 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-sd9mh"] Oct 30 20:03:24.161842 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:24.161809 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ae7c1e-5173-4ed5-828a-8627fc7e2920" path="/var/lib/kubelet/pods/b6ae7c1e-5173-4ed5-828a-8627fc7e2920/volumes" Oct 30 20:03:24.374651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:24.374613 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:24.379458 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:24.379435 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:24.518227 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:24.518201 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:24.519244 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:24.519219 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:25.158381 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:25.158350 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:25.286384 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:25.286354 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.439885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.418291 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7"] Oct 30 20:03:26.439885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.419035 6514 cpu_manager.go:414] "containerMap: removing container" podUID="b6ae7c1e-5173-4ed5-828a-8627fc7e2920" containerName="maas-api" Oct 30 20:03:26.439885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.419062 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ae7c1e-5173-4ed5-828a-8627fc7e2920" containerName="maas-api" Oct 30 20:03:26.439885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.419192 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ae7c1e-5173-4ed5-828a-8627fc7e2920" containerName="maas-api" Oct 30 20:03:26.439885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.424277 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.442068 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.442042 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.446262 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.446236 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7"] Oct 30 20:03:26.448834 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.448810 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc"] Oct 30 20:03:26.449505 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.449482 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-58847295\"" Oct 30 20:03:26.456410 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.456382 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.464142 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.463478 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc"] Oct 30 20:03:26.490723 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.490697 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-58847295-dockercfg-tkhpf\"" Oct 30 20:03:26.497436 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.497406 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph"] Oct 30 20:03:26.501783 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.501760 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.512856 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.512825 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-58847295-serving-cert\"" Oct 30 20:03:26.516645 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.516619 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph"] Oct 30 20:03:26.531890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.531844 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjdj\" (UniqueName: \"kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj\") pod \"payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.532017 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.531976 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.532094 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.532046 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqlg\" (UniqueName: \"kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg\") pod \"payload-processing-e2e-ait-58847295-865b6f598b-q29b7\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.532094 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.532087 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-58847295-865b6f598b-q29b7\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.555757 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.555734 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.633087 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633052 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.633087 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633089 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4c8\" (UniqueName: \"kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.633285 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633242 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjdj\" (UniqueName: \"kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj\") pod \"payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.633394 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633371 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.633542 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633440 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.633542 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633494 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqlg\" (UniqueName: \"kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg\") pod \"payload-processing-e2e-ait-58847295-865b6f598b-q29b7\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.633542 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633538 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-58847295-865b6f598b-q29b7\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.633713 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.633582 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.634254 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.634235 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.634254 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.634245 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume\") pod \"payload-processing-e2e-ait-58847295-865b6f598b-q29b7\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.642802 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.642777 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqlg\" (UniqueName: \"kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg\") pod \"payload-processing-e2e-ait-58847295-865b6f598b-q29b7\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.643018 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.642993 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjdj\" (UniqueName: \"kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj\") pod \"payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.671644 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.671619 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.737647 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.737546 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.737842 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.737652 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.737842 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.737700 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4c8\" (UniqueName: \"kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.738026 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.737842 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.738534 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.738500 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.740609 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.740576 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.741174 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.741137 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.747087 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.747066 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4c8\" (UniqueName: \"kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8\") pod \"maas-api-e2e-ait-58847295-65dd998775-69zph\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:26.770916 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.770894 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:26.779178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.779156 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:26.814306 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.814224 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.820899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:26.820152 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:27.052395 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.052360 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7"] Oct 30 20:03:27.054190 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:27.054163 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded75f458_4fde_41cb_b7bc_b6183d69e9e9.slice/crio-ffbc53ab7f88b7cfdea1711340f48bc5b718a8c9373193779d4fe8402a9943c8 WatchSource:0}: Error finding container ffbc53ab7f88b7cfdea1711340f48bc5b718a8c9373193779d4fe8402a9943c8: Status 404 returned error can't find the container with id ffbc53ab7f88b7cfdea1711340f48bc5b718a8c9373193779d4fe8402a9943c8 Oct 30 20:03:27.065241 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.065215 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc"] Oct 30 20:03:27.077394 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:27.077356 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea4cfd_93bf_4c5a_8275_cdc9703ceb06.slice/crio-8e866eaa51991dd6e9be731b6a8ca57c1bd4c6784f1526006cacaa93303b1d9f WatchSource:0}: Error finding container 8e866eaa51991dd6e9be731b6a8ca57c1bd4c6784f1526006cacaa93303b1d9f: Status 404 returned error can't find the container with id 8e866eaa51991dd6e9be731b6a8ca57c1bd4c6784f1526006cacaa93303b1d9f Oct 30 20:03:27.123586 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.123501 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph"] Oct 30 20:03:27.125163 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:27.125123 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1837f1b5_a2e3_427b_b240_93379022fbc6.slice/crio-72e40560fb25b2b4be714f1374013c69e8116e8f414740da5c29ac9cdb37c7b9 WatchSource:0}: Error finding container 72e40560fb25b2b4be714f1374013c69e8116e8f414740da5c29ac9cdb37c7b9: Status 404 returned error can't find the container with id 72e40560fb25b2b4be714f1374013c69e8116e8f414740da5c29ac9cdb37c7b9 Oct 30 20:03:27.474228 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.474197 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:27.534684 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.534648 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" event={"ID":"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06","Type":"ContainerStarted","Data":"1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2"} Oct 30 20:03:27.534910 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.534708 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" event={"ID":"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06","Type":"ContainerStarted","Data":"8e866eaa51991dd6e9be731b6a8ca57c1bd4c6784f1526006cacaa93303b1d9f"} Oct 30 20:03:27.534910 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.534837 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:27.535903 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.535855 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" event={"ID":"1837f1b5-a2e3-427b-b240-93379022fbc6","Type":"ContainerStarted","Data":"72e40560fb25b2b4be714f1374013c69e8116e8f414740da5c29ac9cdb37c7b9"} Oct 30 20:03:27.537259 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.537239 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" event={"ID":"ed75f458-4fde-41cb-b7bc-b6183d69e9e9","Type":"ContainerStarted","Data":"84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91"} Oct 30 20:03:27.537353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.537276 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" event={"ID":"ed75f458-4fde-41cb-b7bc-b6183d69e9e9","Type":"ContainerStarted","Data":"ffbc53ab7f88b7cfdea1711340f48bc5b718a8c9373193779d4fe8402a9943c8"} Oct 30 20:03:27.537411 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.537378 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:27.554300 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.554260 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" podStartSLOduration=1.554249205 podStartE2EDuration="1.554249205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.552386235 +0000 UTC m=+2777.510205423" watchObservedRunningTime="2026-10-30 20:03:27.554249205 +0000 UTC m=+2777.512068394" Oct 30 20:03:27.572966 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:27.572927 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" podStartSLOduration=1.572914835 podStartE2EDuration="1.572914835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.57115732 +0000 UTC m=+2777.528976510" watchObservedRunningTime="2026-10-30 20:03:27.572914835 +0000 UTC m=+2777.530734021" Oct 30 20:03:28.390644 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:28.390605 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.544114 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:28.544080 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" event={"ID":"1837f1b5-a2e3-427b-b240-93379022fbc6","Type":"ContainerStarted","Data":"7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293"} Oct 30 20:03:28.544504 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:28.544165 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:28.566341 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:28.566295 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" podStartSLOduration=2.025236523 podStartE2EDuration="2.56628332s" totalImagesPullingTime="541.046797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:28.565629467 +0000 UTC m=+2778.523448657" watchObservedRunningTime="2026-10-30 20:03:28.56628332 +0000 UTC m=+2778.524102509" Oct 30 20:03:29.063681 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:29.063650 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:03:29.171233 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:29.171199 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.265223 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:30.265190 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:31.077608 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:31.077571 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:31.908245 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:31.908213 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:32.754803 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:32.754768 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.575908 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:33.575855 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.372507 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:34.372467 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.556728 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:34.556699 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:35.150453 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:35.150422 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.085516 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:36.085481 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.952687 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:36.952657 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.767041 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:37.767008 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.547132 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:38.547104 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:38.547633 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:38.547176 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:38.551758 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:38.551706 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.329323 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.329289 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.520298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.520262 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 20:03:39.520482 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.520320 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 20:03:39.611247 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.611170 6514 generic.go:356] "Generic (PLEG): container finished" podID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerID="71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4" exitCode=0 Oct 30 20:03:39.611247 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.611218 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" event={"ID":"9fc77c52-2cc6-4282-bde2-3cb45085f959","Type":"ContainerDied","Data":"71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4"} Oct 30 20:03:39.611668 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.611587 6514 scope.go:122] "RemoveContainer" containerID="71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4" Oct 30 20:03:39.612756 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.612725 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:39.612814 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.612797 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:39.612850 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:39.612830 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:40.063242 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:40.063211 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.625404 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:40.625369 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" event={"ID":"9fc77c52-2cc6-4282-bde2-3cb45085f959","Type":"ContainerStarted","Data":"08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b"} Oct 30 20:03:40.860661 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:40.860628 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:41.375154 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:41.375113 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:41.375354 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:41.375327 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 20:03:41.375415 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:41.375368 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 20:03:42.013887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:42.013799 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:42.374929 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:42.374837 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 20:03:42.374929 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:42.374920 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 20:03:42.764101 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:42.764069 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:43.375766 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:43.375731 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 20:03:43.376181 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:43.375789 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 20:03:43.556686 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:43.556647 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.263601 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:44.263562 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.375539 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:44.375503 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 20:03:44.375753 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:44.375578 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 20:03:45.056206 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:45.056171 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.375288 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:45.375199 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 20:03:45.375288 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:45.375256 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 20:03:45.836284 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:45.836252 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.379271 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:46.379239 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:46.379688 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:46.379534 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:46.380283 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:46.380265 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:03:46.646544 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:46.646458 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.350836 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:47.350805 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.047759 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:48.047725 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.757187 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:48.757155 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.452850 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:49.452822 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:50.147088 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:50.147056 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:50.888700 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:03:50.888661 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:03:50.944815 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:50.944787 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.109410 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:51.109377 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7"] Oct 30 20:03:51.123733 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:51.123702 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc"] Oct 30 20:03:51.208123 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:51.208090 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.314145 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:51.314113 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.656255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:51.656154 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.161735 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.161651 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph"] Oct 30 20:03:52.162208 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.161854 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" podUID="1837f1b5-a2e3-427b-b240-93379022fbc6" containerName="maas-api" containerID="cri-o://7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293" gracePeriod=30 Oct 30 20:03:52.222184 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.222153 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.382166 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.382137 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.413648 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.413595 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:52.432992 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.432633 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca\") pod \"1837f1b5-a2e3-427b-b240-93379022fbc6\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " Oct 30 20:03:52.432992 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.432780 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls\") pod \"1837f1b5-a2e3-427b-b240-93379022fbc6\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " Oct 30 20:03:52.432992 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.432823 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8\" (UniqueName: \"kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8\") pod \"1837f1b5-a2e3-427b-b240-93379022fbc6\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " Oct 30 20:03:52.432992 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.432848 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs\") pod \"1837f1b5-a2e3-427b-b240-93379022fbc6\" (UID: \"1837f1b5-a2e3-427b-b240-93379022fbc6\") " Oct 30 20:03:52.433764 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.433732 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca" pod "1837f1b5-a2e3-427b-b240-93379022fbc6" (UID: "1837f1b5-a2e3-427b-b240-93379022fbc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:52.436434 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.436404 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8" pod "1837f1b5-a2e3-427b-b240-93379022fbc6" (UID: "1837f1b5-a2e3-427b-b240-93379022fbc6"). InnerVolumeSpecName "kube-api-access-vf4c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:52.437098 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.437074 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls" pod "1837f1b5-a2e3-427b-b240-93379022fbc6" (UID: "1837f1b5-a2e3-427b-b240-93379022fbc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:52.437596 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.437569 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs" pod "1837f1b5-a2e3-427b-b240-93379022fbc6" (UID: "1837f1b5-a2e3-427b-b240-93379022fbc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:52.485540 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.485501 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.534135 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.534104 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1837f1b5-a2e3-427b-b240-93379022fbc6-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:52.534135 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.534138 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:52.534380 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.534155 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4c8\" (UniqueName: \"kubernetes.io/projected/1837f1b5-a2e3-427b-b240-93379022fbc6-kube-api-access-vf4c8\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:52.534380 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.534171 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1837f1b5-a2e3-427b-b240-93379022fbc6-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:52.692559 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.692524 6514 generic.go:356] "Generic (PLEG): container finished" podID="1837f1b5-a2e3-427b-b240-93379022fbc6" containerID="7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293" exitCode=0 Oct 30 20:03:52.692736 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.692571 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" Oct 30 20:03:52.692736 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.692613 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" event={"ID":"1837f1b5-a2e3-427b-b240-93379022fbc6","Type":"ContainerDied","Data":"7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293"} Oct 30 20:03:52.692736 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.692676 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph" event={"ID":"1837f1b5-a2e3-427b-b240-93379022fbc6","Type":"ContainerDied","Data":"72e40560fb25b2b4be714f1374013c69e8116e8f414740da5c29ac9cdb37c7b9"} Oct 30 20:03:52.692736 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.692706 6514 scope.go:122] "RemoveContainer" containerID="7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293" Oct 30 20:03:52.702716 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.702699 6514 scope.go:122] "RemoveContainer" containerID="7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293" Oct 30 20:03:52.703003 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:03:52.702981 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293\": container with ID starting with 7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293 not found: ID does not exist" containerID="7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293" Oct 30 20:03:52.703143 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.703012 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293"} err="failed to get container status \"7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293\": rpc error: code = NotFound desc = could not find container \"7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293\": container with ID starting with 7ba88db9476242b299b263039dc37fae0b712ef28b766c15e3ba55da4680a293 not found: ID does not exist" Oct 30 20:03:52.719377 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.719354 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph"] Oct 30 20:03:52.723089 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:52.723068 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-58847295-65dd998775-69zph"] Oct 30 20:03:53.830787 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:53.830745 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:03:54.157208 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:54.157130 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1837f1b5-a2e3-427b-b240-93379022fbc6" path="/var/lib/kubelet/pods/1837f1b5-a2e3-427b-b240-93379022fbc6/volumes" Oct 30 20:03:56.110580 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.110534 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" podUID="ed75f458-4fde-41cb-b7bc-b6183d69e9e9" containerName="payload-processing" containerID="cri-o://84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91" gracePeriod=25 Oct 30 20:03:56.124032 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.124001 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" podUID="cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" containerName="payload-pre-processing" containerID="cri-o://1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2" gracePeriod=25 Oct 30 20:03:56.374172 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.374149 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:56.400848 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.400824 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:56.471312 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.471287 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj\" (UniqueName: \"kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj\") pod \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " Oct 30 20:03:56.471471 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.471323 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg\" (UniqueName: \"kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg\") pod \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " Oct 30 20:03:56.471471 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.471429 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume\") pod \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\" (UID: \"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06\") " Oct 30 20:03:56.471556 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.471487 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume\") pod \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\" (UID: \"ed75f458-4fde-41cb-b7bc-b6183d69e9e9\") " Oct 30 20:03:56.471893 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.471849 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume" pod "cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" (UID: "cbea4cfd-93bf-4c5a-8275-cdc9703ceb06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:56.471955 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.471904 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume" pod "ed75f458-4fde-41cb-b7bc-b6183d69e9e9" (UID: "ed75f458-4fde-41cb-b7bc-b6183d69e9e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:56.473541 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.473517 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj" pod "cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" (UID: "cbea4cfd-93bf-4c5a-8275-cdc9703ceb06"). InnerVolumeSpecName "kube-api-access-mdjdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:56.473597 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.473552 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg" pod "ed75f458-4fde-41cb-b7bc-b6183d69e9e9" (UID: "ed75f458-4fde-41cb-b7bc-b6183d69e9e9"). InnerVolumeSpecName "kube-api-access-ztqlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:56.573172 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.573137 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdjdj\" (UniqueName: \"kubernetes.io/projected/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-kube-api-access-mdjdj\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:56.573172 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.573168 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqlg\" (UniqueName: \"kubernetes.io/projected/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-kube-api-access-ztqlg\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:56.573172 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.573179 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:56.573403 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.573188 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed75f458-4fde-41cb-b7bc-b6183d69e9e9-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:56.713900 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.713854 6514 generic.go:356] "Generic (PLEG): container finished" podID="ed75f458-4fde-41cb-b7bc-b6183d69e9e9" containerID="84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91" exitCode=0 Oct 30 20:03:56.714071 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.713911 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" Oct 30 20:03:56.714071 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.713954 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" event={"ID":"ed75f458-4fde-41cb-b7bc-b6183d69e9e9","Type":"ContainerDied","Data":"84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91"} Oct 30 20:03:56.714071 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.714002 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7" event={"ID":"ed75f458-4fde-41cb-b7bc-b6183d69e9e9","Type":"ContainerDied","Data":"ffbc53ab7f88b7cfdea1711340f48bc5b718a8c9373193779d4fe8402a9943c8"} Oct 30 20:03:56.714071 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.714026 6514 scope.go:122] "RemoveContainer" containerID="84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91" Oct 30 20:03:56.715332 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.715302 6514 generic.go:356] "Generic (PLEG): container finished" podID="cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" containerID="1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2" exitCode=0 Oct 30 20:03:56.715468 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.715333 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" event={"ID":"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06","Type":"ContainerDied","Data":"1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2"} Oct 30 20:03:56.715468 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.715383 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" event={"ID":"cbea4cfd-93bf-4c5a-8275-cdc9703ceb06","Type":"ContainerDied","Data":"8e866eaa51991dd6e9be731b6a8ca57c1bd4c6784f1526006cacaa93303b1d9f"} Oct 30 20:03:56.715468 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.715345 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc" Oct 30 20:03:56.723076 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.723007 6514 scope.go:122] "RemoveContainer" containerID="84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91" Oct 30 20:03:56.723342 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:03:56.723314 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91\": container with ID starting with 84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91 not found: ID does not exist" containerID="84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91" Oct 30 20:03:56.723443 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.723348 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91"} err="failed to get container status \"84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91\": rpc error: code = NotFound desc = could not find container \"84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91\": container with ID starting with 84ce60a24974cc2eaa06006fce899a051dd307761b0a8a9f83d8f1b6ab420a91 not found: ID does not exist" Oct 30 20:03:56.723443 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.723362 6514 scope.go:122] "RemoveContainer" containerID="1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2" Oct 30 20:03:56.731624 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.731607 6514 scope.go:122] "RemoveContainer" containerID="1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2" Oct 30 20:03:56.731924 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:03:56.731903 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2\": container with ID starting with 1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2 not found: ID does not exist" containerID="1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2" Oct 30 20:03:56.732005 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.731931 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2"} err="failed to get container status \"1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2\": rpc error: code = NotFound desc = could not find container \"1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2\": container with ID starting with 1705cabc9a17690478727cfd4e8e3baf634feb2e7202cae5d568c0f0cba417c2 not found: ID does not exist" Oct 30 20:03:56.742080 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.742058 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc"] Oct 30 20:03:56.746547 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.746523 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-58847295-7cbc7fcfc5-7srtc"] Oct 30 20:03:56.762579 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.762557 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7"] Oct 30 20:03:56.766118 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:56.766098 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-58847295-865b6f598b-q29b7"] Oct 30 20:03:57.359001 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:57.358968 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26"] Oct 30 20:03:57.359502 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:57.359187 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" containerID="cri-o://08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b" gracePeriod=30 Oct 30 20:03:57.417321 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:57.417251 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.157478 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:58.157446 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" path="/var/lib/kubelet/pods/cbea4cfd-93bf-4c5a-8275-cdc9703ceb06/volumes" Oct 30 20:03:58.157843 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:58.157829 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed75f458-4fde-41cb-b7bc-b6183d69e9e9" path="/var/lib/kubelet/pods/ed75f458-4fde-41cb-b7bc-b6183d69e9e9/volumes" Oct 30 20:03:58.680875 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:58.680837 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.066319 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066236 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m"] Oct 30 20:03:59.066670 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066655 6514 cpu_manager.go:414] "containerMap: removing container" podUID="cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" containerName="payload-pre-processing" Oct 30 20:03:59.066749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066670 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" containerName="payload-pre-processing" Oct 30 20:03:59.066749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066697 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ed75f458-4fde-41cb-b7bc-b6183d69e9e9" containerName="payload-processing" Oct 30 20:03:59.066749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066703 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed75f458-4fde-41cb-b7bc-b6183d69e9e9" containerName="payload-processing" Oct 30 20:03:59.066749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066714 6514 cpu_manager.go:414] "containerMap: removing container" podUID="1837f1b5-a2e3-427b-b240-93379022fbc6" containerName="maas-api" Oct 30 20:03:59.066749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066719 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1837f1b5-a2e3-427b-b240-93379022fbc6" containerName="maas-api" Oct 30 20:03:59.067072 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066797 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed75f458-4fde-41cb-b7bc-b6183d69e9e9" containerName="payload-processing" Oct 30 20:03:59.067072 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066807 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="1837f1b5-a2e3-427b-b240-93379022fbc6" containerName="maas-api" Oct 30 20:03:59.067072 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.066816 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbea4cfd-93bf-4c5a-8275-cdc9703ceb06" containerName="payload-pre-processing" Oct 30 20:03:59.071660 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.071637 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.081475 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.081449 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-64f2bb-openshift-default-dockercfg-mx272\"" Oct 30 20:03:59.085141 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.085116 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m"] Oct 30 20:03:59.137616 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.137581 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.199422 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199387 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199599 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199447 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199599 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199492 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199599 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199513 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199717 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199587 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199717 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199660 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199788 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199720 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199857 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199837 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44h6x\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.199925 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.199891 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.273206 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.273168 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.300985 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.300955 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301122 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.300988 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44h6x\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301122 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301016 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301229 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301120 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301229 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301173 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301229 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301217 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301401 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301240 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301401 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301270 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301401 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301305 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301570 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301431 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301570 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301516 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301676 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301608 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.301768 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.301746 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.302102 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.302080 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.303367 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.303341 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.303750 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.303733 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.310115 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.310090 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.310239 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.310221 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44h6x\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x\") pod \"e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.387079 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.387006 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:03:59.665536 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.665506 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m"] Oct 30 20:03:59.666994 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:03:59.666968 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97519e16_b1a9_4c00_aa01_c648ff56de68.slice/crio-7a28abcfce33d052fe936c7a54aacebc2e606c1177ad5e593447f8c0ad58db85 WatchSource:0}: Error finding container 7a28abcfce33d052fe936c7a54aacebc2e606c1177ad5e593447f8c0ad58db85: Status 404 returned error can't find the container with id 7a28abcfce33d052fe936c7a54aacebc2e606c1177ad5e593447f8c0ad58db85 Oct 30 20:03:59.669517 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.669484 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:59.669616 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.669571 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:59.669683 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.669627 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:59.745682 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.745654 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" event={"ID":"97519e16-b1a9-4c00-aa01-c648ff56de68","Type":"ContainerStarted","Data":"2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d"} Oct 30 20:03:59.746019 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.745705 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" event={"ID":"97519e16-b1a9-4c00-aa01-c648ff56de68","Type":"ContainerStarted","Data":"7a28abcfce33d052fe936c7a54aacebc2e606c1177ad5e593447f8c0ad58db85"} Oct 30 20:03:59.770048 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:03:59.769998 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podStartSLOduration=0.769983855 podStartE2EDuration="769.983855ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:59.768617697 +0000 UTC m=+2809.726436886" watchObservedRunningTime="2026-10-30 20:03:59.769983855 +0000 UTC m=+2809.727803044" Oct 30 20:04:00.387238 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:00.387198 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:00.392201 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:00.392175 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:00.751820 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:00.751791 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:00.752640 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:00.752619 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:01.381594 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:01.381559 6514 patch_prober.go:29] interesting pod/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:04:01.381794 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:01.381618 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:04:02.011448 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.011409 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.140337 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.140301 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.612134 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.612109 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:04:02.738818 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.738785 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739012 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.738828 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739012 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.738998 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739129 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739073 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739175 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739125 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739299 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739277 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739365 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739317 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.739365 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739352 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739480 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739370 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.739480 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739416 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739587 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739463 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:02.739587 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739549 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.739708 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739476 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy\") pod \"9fc77c52-2cc6-4282-bde2-3cb45085f959\" (UID: \"9fc77c52-2cc6-4282-bde2-3cb45085f959\") " Oct 30 20:04:02.739708 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739614 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.740073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.739982 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.740073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.740009 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.740073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.740022 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc77c52-2cc6-4282-bde2-3cb45085f959-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.740073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.740037 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.740073 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.740052 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.741493 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.741460 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:02.741597 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.741516 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.741644 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.741606 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "kube-api-access-dgvf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:02.741758 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.741744 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token" pod "9fc77c52-2cc6-4282-bde2-3cb45085f959" (UID: "9fc77c52-2cc6-4282-bde2-3cb45085f959"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:02.764839 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.764813 6514 generic.go:356] "Generic (PLEG): container finished" podID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerID="08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b" exitCode=0 Oct 30 20:04:02.765021 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.764884 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" Oct 30 20:04:02.765021 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.764906 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" event={"ID":"9fc77c52-2cc6-4282-bde2-3cb45085f959","Type":"ContainerDied","Data":"08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b"} Oct 30 20:04:02.765021 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.764971 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26" event={"ID":"9fc77c52-2cc6-4282-bde2-3cb45085f959","Type":"ContainerDied","Data":"c658b965305e57cbad03e230f7731d7478587dce09ef3f5a7438c4c3f95fd162"} Oct 30 20:04:02.765021 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.765001 6514 scope.go:122] "RemoveContainer" containerID="08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b" Oct 30 20:04:02.774732 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.774716 6514 scope.go:122] "RemoveContainer" containerID="71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4" Oct 30 20:04:02.783202 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.783185 6514 scope.go:122] "RemoveContainer" containerID="08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b" Oct 30 20:04:02.783443 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:04:02.783428 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b\": container with ID starting with 08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b not found: ID does not exist" containerID="08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b" Oct 30 20:04:02.783494 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.783453 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b"} err="failed to get container status \"08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b\": rpc error: code = NotFound desc = could not find container \"08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b\": container with ID starting with 08632c7f19c4a92256f2a59a998f9951f6934eda3b399c230474fec08f49573b not found: ID does not exist" Oct 30 20:04:02.783494 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.783469 6514 scope.go:122] "RemoveContainer" containerID="71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4" Oct 30 20:04:02.783682 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:04:02.783663 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4\": container with ID starting with 71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4 not found: ID does not exist" containerID="71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4" Oct 30 20:04:02.783723 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.783695 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4"} err="failed to get container status \"71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4\": rpc error: code = NotFound desc = could not find container \"71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4\": container with ID starting with 71ad61c72aaa17db0c1e9c7ae1893e2e2f44730616fd9fd48c9226fb199e52b4 not found: ID does not exist" Oct 30 20:04:02.788089 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.788060 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26"] Oct 30 20:04:02.792451 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.792427 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-58847295-openshift-default-8f844cdfc-m4s26"] Oct 30 20:04:02.841221 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.841185 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.841221 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.841222 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.841434 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.841241 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.841434 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:02.841256 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvf9\" (UniqueName: \"kubernetes.io/projected/9fc77c52-2cc6-4282-bde2-3cb45085f959-kube-api-access-dgvf9\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:03.188685 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.188648 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv"] Oct 30 20:04:03.190107 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.190080 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" Oct 30 20:04:03.190107 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.190106 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" Oct 30 20:04:03.190420 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.190138 6514 cpu_manager.go:414] "containerMap: removing container" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" Oct 30 20:04:03.190420 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.190146 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" Oct 30 20:04:03.190420 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.190239 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" Oct 30 20:04:03.190420 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.190255 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" containerName="istio-proxy" Oct 30 20:04:03.195232 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.195209 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.206026 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.206004 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-64f2bb-serving-cert\"" Oct 30 20:04:03.207614 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.207588 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv"] Oct 30 20:04:03.226399 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.226365 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.245735 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.245705 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfzz\" (UniqueName: \"kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.245897 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.245757 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.245897 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.245839 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.246004 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.245943 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.249977 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.249955 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h"] Oct 30 20:04:03.253694 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.253677 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.262272 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.262248 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-64f2bb\"" Oct 30 20:04:03.265277 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.265257 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h"] Oct 30 20:04:03.318127 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.318101 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh"] Oct 30 20:04:03.321726 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.321707 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.331303 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.331277 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-64f2bb-dockercfg-9bqfl\"" Oct 30 20:04:03.335522 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.335500 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh"] Oct 30 20:04:03.347080 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.346848 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.347080 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.346990 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.347080 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.347042 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.347306 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.347127 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.347306 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.347223 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w224\" (UniqueName: \"kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224\") pod \"payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.347306 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.347287 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfzz\" (UniqueName: \"kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.348137 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.348112 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.349551 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.349524 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.349625 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.349567 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.353595 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.353571 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.361279 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.361258 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfzz\" (UniqueName: \"kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz\") pod \"maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.448435 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.448339 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w224\" (UniqueName: \"kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224\") pod \"payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.448435 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.448402 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlp2\" (UniqueName: \"kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2\") pod \"payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.448655 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.448497 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.448655 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.448545 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.449285 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.449258 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.457895 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.457851 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w224\" (UniqueName: \"kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224\") pod \"payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.511679 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.511649 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:03.550213 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.550179 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlp2\" (UniqueName: \"kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2\") pod \"payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.550350 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.550258 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.551013 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.550987 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.559651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.559620 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlp2\" (UniqueName: \"kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2\") pod \"payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.567404 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.567378 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:03.637124 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.636813 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:03.789952 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.789850 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv"] Oct 30 20:04:03.792040 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:04:03.792010 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9e9715_f7f1_4407_9a84_f80148af69bc.slice/crio-0931c83f68b5bb1ee844d92a895e1b92b42040b42ea99e7334a32429d9ac486e WatchSource:0}: Error finding container 0931c83f68b5bb1ee844d92a895e1b92b42040b42ea99e7334a32429d9ac486e: Status 404 returned error can't find the container with id 0931c83f68b5bb1ee844d92a895e1b92b42040b42ea99e7334a32429d9ac486e Oct 30 20:04:03.850488 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.850455 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h"] Oct 30 20:04:03.851325 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:04:03.851291 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffad5dec_cb6f_419e_817d_ab985fb6d823.slice/crio-85942b10bb80bba83eafd506f84c79913042bce97c74d0ff9b9b4e5470f1d6f9 WatchSource:0}: Error finding container 85942b10bb80bba83eafd506f84c79913042bce97c74d0ff9b9b4e5470f1d6f9: Status 404 returned error can't find the container with id 85942b10bb80bba83eafd506f84c79913042bce97c74d0ff9b9b4e5470f1d6f9 Oct 30 20:04:03.919968 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:04:03.919933 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b0a8a2_ecb8_457f_8c32_5756c3e500d2.slice/crio-52fd6e392d77ebeca8686b40c495790d1504ffe7da0f3a4cd073a2f9b691fed0 WatchSource:0}: Error finding container 52fd6e392d77ebeca8686b40c495790d1504ffe7da0f3a4cd073a2f9b691fed0: Status 404 returned error can't find the container with id 52fd6e392d77ebeca8686b40c495790d1504ffe7da0f3a4cd073a2f9b691fed0 Oct 30 20:04:03.920213 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:03.920187 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh"] Oct 30 20:04:04.074938 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.074772 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.157378 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.157344 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc77c52-2cc6-4282-bde2-3cb45085f959" path="/var/lib/kubelet/pods/9fc77c52-2cc6-4282-bde2-3cb45085f959/volumes" Oct 30 20:04:04.783703 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.783668 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" event={"ID":"59b0a8a2-ecb8-457f-8c32-5756c3e500d2","Type":"ContainerStarted","Data":"0b1551199a9e8c5376c8f661a8ff045c3ffdd2d0d780b01129fca3f9018ea417"} Oct 30 20:04:04.784480 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.783723 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" event={"ID":"59b0a8a2-ecb8-457f-8c32-5756c3e500d2","Type":"ContainerStarted","Data":"52fd6e392d77ebeca8686b40c495790d1504ffe7da0f3a4cd073a2f9b691fed0"} Oct 30 20:04:04.784480 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.783750 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:04.785376 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.785342 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" event={"ID":"4f9e9715-f7f1-4407-9a84-f80148af69bc","Type":"ContainerStarted","Data":"bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a"} Oct 30 20:04:04.785520 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.785400 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" event={"ID":"4f9e9715-f7f1-4407-9a84-f80148af69bc","Type":"ContainerStarted","Data":"0931c83f68b5bb1ee844d92a895e1b92b42040b42ea99e7334a32429d9ac486e"} Oct 30 20:04:04.785520 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.785434 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:04.786922 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.786903 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" event={"ID":"ffad5dec-cb6f-419e-817d-ab985fb6d823","Type":"ContainerStarted","Data":"193e3d064f4fecdde2d394316ba8d3579b11fe4ec029883f67a53fc06f023114"} Oct 30 20:04:04.787001 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.786936 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" event={"ID":"ffad5dec-cb6f-419e-817d-ab985fb6d823","Type":"ContainerStarted","Data":"85942b10bb80bba83eafd506f84c79913042bce97c74d0ff9b9b4e5470f1d6f9"} Oct 30 20:04:04.787001 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.786972 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:04.803250 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.803211 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" podStartSLOduration=1.8032001819999999 podStartE2EDuration="1.803200182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:04.801410062 +0000 UTC m=+2814.759229254" watchObservedRunningTime="2026-10-30 20:04:04.803200182 +0000 UTC m=+2814.761019370" Oct 30 20:04:04.839790 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.839743 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" podStartSLOduration=1.8397308620000001 podStartE2EDuration="1.839730862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:04.838879225 +0000 UTC m=+2814.796698406" watchObservedRunningTime="2026-10-30 20:04:04.839730862 +0000 UTC m=+2814.797550047" Oct 30 20:04:04.840019 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.839992 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" podStartSLOduration=1.289369845 podStartE2EDuration="1.839984519s" totalImagesPullingTime="550.614674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:04.819969414 +0000 UTC m=+2814.777788602" watchObservedRunningTime="2026-10-30 20:04:04.839984519 +0000 UTC m=+2814.797803698" Oct 30 20:04:04.862735 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:04.862708 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.335717 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:05.335680 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:04:05.697902 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:05.696469 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:06.651038 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:06.651004 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.514432 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:07.514387 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.278535 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:08.278507 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.085499 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:09.085453 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.852189 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:09.852156 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.631315 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:10.631276 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.821070 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:10.821042 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:04:11.449584 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:11.449541 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.222440 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:12.222401 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.942586 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:12.942549 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.365200 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:13.365115 6514 scope.go:122] "RemoveContainer" containerID="5546578811ee110cbc79e3db934a161ea47360297956f44d5023ecdcfae22f0e" Oct 30 20:04:13.654194 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:13.654106 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.760412 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:13.760378 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.351812 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.351783 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.877170 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.877136 6514 generic.go:356] "Generic (PLEG): container finished" podID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerID="2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d" exitCode=0 Oct 30 20:04:14.877365 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.877194 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" event={"ID":"97519e16-b1a9-4c00-aa01-c648ff56de68","Type":"ContainerDied","Data":"2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d"} Oct 30 20:04:14.877649 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.877629 6514 scope.go:122] "RemoveContainer" containerID="2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d" Oct 30 20:04:14.879036 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.879001 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:04:14.879132 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.879084 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:04:14.879132 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:14.879124 6514 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:04:15.058935 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:15.058895 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.752413 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:15.752368 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:15.817568 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:15.817453 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:04:15.818147 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:15.818116 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:04:15.863193 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:15.863143 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.885304 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:15.885265 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" event={"ID":"97519e16-b1a9-4c00-aa01-c648ff56de68","Type":"ContainerStarted","Data":"6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942"} Oct 30 20:04:16.388136 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:16.388094 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:16.389175 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:16.389144 6514 patch_prober.go:29] interesting pod/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 20:04:16.389344 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:16.389197 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 20:04:16.675912 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:16.675811 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:17.015890 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:17.014978 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:17.388361 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:17.388284 6514 patch_prober.go:29] interesting pod/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 20:04:17.388361 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:17.388338 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 20:04:17.532302 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:17.532261 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.381012 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:18.380971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.388063 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:18.388035 6514 patch_prober.go:29] interesting pod/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 20:04:18.388191 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:18.388091 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 20:04:19.258207 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:19.258173 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.387853 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:19.387821 6514 patch_prober.go:29] interesting pod/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 20:04:19.388261 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:19.387898 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 20:04:20.161785 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:20.161751 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:20.387574 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:20.387537 6514 patch_prober.go:29] interesting pod/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 20:04:20.387752 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:20.387591 6514 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 20:04:20.882232 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:04:20.882192 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:04:20.954539 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:20.954503 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:21.392442 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:21.392411 6514 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:21.392690 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:21.392671 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:21.393319 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:21.393300 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:04:21.651603 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:21.651523 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:22.363000 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:22.362964 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:23.043979 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:23.043945 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:23.843443 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:23.843414 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:24.451684 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:24.451642 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:25.148558 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:25.148516 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:25.842159 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:25.842123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:26.955604 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:26.955570 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:27.658586 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:27.658551 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:28.349768 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:28.349735 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.046405 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:29.046369 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.756472 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:29.756439 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.549907 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:30.549858 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.256171 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:31.256131 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.963753 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:31.963719 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.657937 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:32.657903 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:33.452955 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:33.452925 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.140841 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:34.140808 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.866308 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:34.866276 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.547299 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:35.547265 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.249267 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:36.249229 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.957068 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:36.957032 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:37.650492 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:37.650458 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:38.446791 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:38.446749 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.247795 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:39.247759 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.107245 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:40.107207 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.747687 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:40.747649 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.446162 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:41.446133 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.157649 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:42.157616 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.941014 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:42.940971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:43.637424 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:43.637386 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:44.340743 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:44.340711 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:45.044285 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:45.044250 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:45.759695 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:45.759663 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.437776 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:46.437719 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:47.196311 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:47.196276 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:47.839632 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:47.839596 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.547136 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:48.547103 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.436284 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:49.436248 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:50.064048 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:50.064016 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:50.743886 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:50.743839 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:50.880809 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:04:50.880772 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:04:51.443230 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:51.443193 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:52.158084 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:52.158051 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:52.953022 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:52.952973 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.711010 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:53.710971 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.565528 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:54.565497 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:55.254060 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:55.254009 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:55.956749 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:55.956715 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:56.652372 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:56.652302 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:57.414121 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:57.414087 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:58.137103 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:58.137066 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:58.848619 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:58.848586 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.549120 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:04:59.549088 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.025484 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:00.025448 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6"] Oct 30 20:05:00.029038 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:00.029012 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-crwz6"] Oct 30 20:05:00.158498 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:00.158463 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c0c871-76ab-4d28-980e-71b7db5209ed" path="/var/lib/kubelet/pods/19c0c871-76ab-4d28-980e-71b7db5209ed/volumes" Oct 30 20:05:00.250422 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:00.250390 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.942236 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:00.942205 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:01.641025 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:01.640993 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:02.344622 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:02.344585 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:03.140353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:03.140321 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:03.808438 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:03.808375 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:04.659808 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:04.659774 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:05.536769 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:05.536734 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:06.240997 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:06.240962 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:06.928052 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:06.928010 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:07.648148 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:07.648109 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.532163 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:08.532129 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.428553 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:09.428503 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:10.200956 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:10.200921 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:10.849023 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:10.848989 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:11.623451 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:11.623407 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:12.449776 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:12.449738 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:13.149255 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:13.149222 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:13.702215 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:13.702178 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:13.848037 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:13.848005 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.555414 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:14.555378 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:15.250722 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:15.250692 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:15.943098 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:15.943069 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:16.657573 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:16.657531 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.396419 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:17.396385 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.512629 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:17.512602 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:18.148149 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:18.148114 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:18.842162 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:18.842130 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:19.534415 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:19.534375 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:20.382110 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:20.382077 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:20.880944 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:05:20.880906 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:05:21.025118 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:21.025079 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7"] Oct 30 20:05:21.028547 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:21.028520 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-grvd7"] Oct 30 20:05:21.056199 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:21.056162 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:21.928249 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:21.928201 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:22.157778 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:22.157708 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0db0ca3-0b9b-4066-a97e-95565077a1d1" path="/var/lib/kubelet/pods/a0db0ca3-0b9b-4066-a97e-95565077a1d1/volumes" Oct 30 20:05:22.655527 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:22.655473 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:23.336585 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:23.336556 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:23.452927 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:23.452898 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.043602 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:24.043566 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.840945 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:24.840891 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:25.639651 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:25.639613 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:26.342461 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:26.342428 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:27.035524 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:27.035475 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:27.734885 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:27.734825 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.526424 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:28.526372 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.138979 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:29.138935 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.935756 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:29.935717 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:30.660374 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:30.660341 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:31.329468 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:31.329434 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.040178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:32.040135 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.757392 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:32.757362 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:33.457043 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:33.457012 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.149685 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:34.149641 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.890019 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:34.889974 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:35.632556 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:35.632523 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:36.534229 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:36.534186 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:37.157396 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:37.157335 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:38.044745 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:38.044716 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:38.757622 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:38.757588 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:39.444003 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:39.443969 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.169653 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:40.169591 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.855643 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:40.855614 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:41.557875 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:41.557831 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.250506 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:42.250474 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:43.053263 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:43.053226 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:43.814008 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:43.813968 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:44.533549 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:44.533515 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:45.310048 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:45.310009 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:46.151491 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:46.151452 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:46.928531 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:46.928500 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.282702 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:47.282625 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh"] Oct 30 20:05:47.309716 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:47.309684 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h"] Oct 30 20:05:47.409191 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:47.409157 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.537171 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:47.537089 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:50.880215 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:05:50.880175 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:05:52.283489 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.283427 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" podUID="59b0a8a2-ecb8-457f-8c32-5756c3e500d2" containerName="payload-processing" containerID="cri-o://0b1551199a9e8c5376c8f661a8ff045c3ffdd2d0d780b01129fca3f9018ea417" gracePeriod=25 Oct 30 20:05:52.310303 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.310259 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" podUID="ffad5dec-cb6f-419e-817d-ab985fb6d823" containerName="payload-pre-processing" containerID="cri-o://193e3d064f4fecdde2d394316ba8d3579b11fe4ec029883f67a53fc06f023114" gracePeriod=25 Oct 30 20:05:52.437107 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.437078 6514 generic.go:356] "Generic (PLEG): container finished" podID="ffad5dec-cb6f-419e-817d-ab985fb6d823" containerID="193e3d064f4fecdde2d394316ba8d3579b11fe4ec029883f67a53fc06f023114" exitCode=0 Oct 30 20:05:52.437249 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.437149 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" event={"ID":"ffad5dec-cb6f-419e-817d-ab985fb6d823","Type":"ContainerDied","Data":"193e3d064f4fecdde2d394316ba8d3579b11fe4ec029883f67a53fc06f023114"} Oct 30 20:05:52.439023 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.439003 6514 generic.go:356] "Generic (PLEG): container finished" podID="59b0a8a2-ecb8-457f-8c32-5756c3e500d2" containerID="0b1551199a9e8c5376c8f661a8ff045c3ffdd2d0d780b01129fca3f9018ea417" exitCode=0 Oct 30 20:05:52.439130 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.439055 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" event={"ID":"59b0a8a2-ecb8-457f-8c32-5756c3e500d2","Type":"ContainerDied","Data":"0b1551199a9e8c5376c8f661a8ff045c3ffdd2d0d780b01129fca3f9018ea417"} Oct 30 20:05:52.561148 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.561124 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:05:52.580352 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.580329 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:05:52.624281 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.624250 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv"] Oct 30 20:05:52.624727 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.624695 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" podUID="4f9e9715-f7f1-4407-9a84-f80148af69bc" containerName="maas-api" containerID="cri-o://bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a" gracePeriod=30 Oct 30 20:05:52.700660 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.700630 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:52.704047 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704021 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume\") pod \"ffad5dec-cb6f-419e-817d-ab985fb6d823\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " Oct 30 20:05:52.704167 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704058 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2\" (UniqueName: \"kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2\") pod \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " Oct 30 20:05:52.704167 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704129 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume\") pod \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\" (UID: \"59b0a8a2-ecb8-457f-8c32-5756c3e500d2\") " Oct 30 20:05:52.704263 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704223 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224\" (UniqueName: \"kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224\") pod \"ffad5dec-cb6f-419e-817d-ab985fb6d823\" (UID: \"ffad5dec-cb6f-419e-817d-ab985fb6d823\") " Oct 30 20:05:52.704400 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704377 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume" pod "ffad5dec-cb6f-419e-817d-ab985fb6d823" (UID: "ffad5dec-cb6f-419e-817d-ab985fb6d823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:52.704507 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704486 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume" pod "59b0a8a2-ecb8-457f-8c32-5756c3e500d2" (UID: "59b0a8a2-ecb8-457f-8c32-5756c3e500d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:52.704668 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704651 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffad5dec-cb6f-419e-817d-ab985fb6d823-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:52.704722 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.704672 6514 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-plugins-config-volume\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:52.706246 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.706224 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224" pod "ffad5dec-cb6f-419e-817d-ab985fb6d823" (UID: "ffad5dec-cb6f-419e-817d-ab985fb6d823"). InnerVolumeSpecName "kube-api-access-6w224". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:52.706328 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.706232 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2" pod "59b0a8a2-ecb8-457f-8c32-5756c3e500d2" (UID: "59b0a8a2-ecb8-457f-8c32-5756c3e500d2"). InnerVolumeSpecName "kube-api-access-zvlp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:52.805799 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.805766 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvlp2\" (UniqueName: \"kubernetes.io/projected/59b0a8a2-ecb8-457f-8c32-5756c3e500d2-kube-api-access-zvlp2\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:52.805953 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.805806 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w224\" (UniqueName: \"kubernetes.io/projected/ffad5dec-cb6f-419e-817d-ab985fb6d823-kube-api-access-6w224\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:52.852231 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.852202 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:52.882200 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.882173 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:05:52.962565 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:52.962534 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.008720 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.008691 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca\") pod \"4f9e9715-f7f1-4407-9a84-f80148af69bc\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " Oct 30 20:05:53.008895 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.008729 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs\") pod \"4f9e9715-f7f1-4407-9a84-f80148af69bc\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " Oct 30 20:05:53.008895 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.008751 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz\" (UniqueName: \"kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz\") pod \"4f9e9715-f7f1-4407-9a84-f80148af69bc\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " Oct 30 20:05:53.008895 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.008813 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls\") pod \"4f9e9715-f7f1-4407-9a84-f80148af69bc\" (UID: \"4f9e9715-f7f1-4407-9a84-f80148af69bc\") " Oct 30 20:05:53.009149 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.009124 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca" pod "4f9e9715-f7f1-4407-9a84-f80148af69bc" (UID: "4f9e9715-f7f1-4407-9a84-f80148af69bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:53.009307 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.009288 6514 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9e9715-f7f1-4407-9a84-f80148af69bc-openshift-service-ca\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:53.011063 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.011023 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz" pod "4f9e9715-f7f1-4407-9a84-f80148af69bc" (UID: "4f9e9715-f7f1-4407-9a84-f80148af69bc"). InnerVolumeSpecName "kube-api-access-khfzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:53.011063 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.011039 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs" pod "4f9e9715-f7f1-4407-9a84-f80148af69bc" (UID: "4f9e9715-f7f1-4407-9a84-f80148af69bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:53.011204 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.011088 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls" pod "4f9e9715-f7f1-4407-9a84-f80148af69bc" (UID: "4f9e9715-f7f1-4407-9a84-f80148af69bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:53.110543 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.110479 6514 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-metrics-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:53.110543 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.110503 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khfzz\" (UniqueName: \"kubernetes.io/projected/4f9e9715-f7f1-4407-9a84-f80148af69bc-kube-api-access-khfzz\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:53.110543 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.110513 6514 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f9e9715-f7f1-4407-9a84-f80148af69bc-maas-api-tls\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:53.445937 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.445904 6514 generic.go:356] "Generic (PLEG): container finished" podID="4f9e9715-f7f1-4407-9a84-f80148af69bc" containerID="bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a" exitCode=0 Oct 30 20:05:53.446353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.445950 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" Oct 30 20:05:53.446353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.445988 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" event={"ID":"4f9e9715-f7f1-4407-9a84-f80148af69bc","Type":"ContainerDied","Data":"bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a"} Oct 30 20:05:53.446353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.446051 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv" event={"ID":"4f9e9715-f7f1-4407-9a84-f80148af69bc","Type":"ContainerDied","Data":"0931c83f68b5bb1ee844d92a895e1b92b42040b42ea99e7334a32429d9ac486e"} Oct 30 20:05:53.446353 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.446077 6514 scope.go:122] "RemoveContainer" containerID="bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a" Oct 30 20:05:53.447721 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.447699 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" Oct 30 20:05:53.447843 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.447727 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h" event={"ID":"ffad5dec-cb6f-419e-817d-ab985fb6d823","Type":"ContainerDied","Data":"85942b10bb80bba83eafd506f84c79913042bce97c74d0ff9b9b4e5470f1d6f9"} Oct 30 20:05:53.449377 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.449356 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" Oct 30 20:05:53.449609 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.449355 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh" event={"ID":"59b0a8a2-ecb8-457f-8c32-5756c3e500d2","Type":"ContainerDied","Data":"52fd6e392d77ebeca8686b40c495790d1504ffe7da0f3a4cd073a2f9b691fed0"} Oct 30 20:05:53.458060 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.458046 6514 scope.go:122] "RemoveContainer" containerID="bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a" Oct 30 20:05:53.458323 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:05:53.458300 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a\": container with ID starting with bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a not found: ID does not exist" containerID="bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a" Oct 30 20:05:53.458439 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.458336 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a"} err="failed to get container status \"bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a\": rpc error: code = NotFound desc = could not find container \"bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a\": container with ID starting with bbaad87fd0ae5d35a8dfc300264250c64d4427c78ce3e5d21ab8a92842353b8a not found: ID does not exist" Oct 30 20:05:53.458439 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.458359 6514 scope.go:122] "RemoveContainer" containerID="193e3d064f4fecdde2d394316ba8d3579b11fe4ec029883f67a53fc06f023114" Oct 30 20:05:53.461918 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.461899 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.467351 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.467328 6514 scope.go:122] "RemoveContainer" containerID="0b1551199a9e8c5376c8f661a8ff045c3ffdd2d0d780b01129fca3f9018ea417" Oct 30 20:05:53.482032 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.475926 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv"] Oct 30 20:05:53.485276 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.485254 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-64f2bb-5cb4c89f74-tvxlv"] Oct 30 20:05:53.496233 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.496213 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh"] Oct 30 20:05:53.498759 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.498738 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-64f2bb-5cbb5f6456-x96sh"] Oct 30 20:05:53.508613 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.508595 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h"] Oct 30 20:05:53.514499 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:53.514480 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-64f2bb-7dbf5f4555-gns4h"] Oct 30 20:05:54.158105 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:54.158075 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9e9715-f7f1-4407-9a84-f80148af69bc" path="/var/lib/kubelet/pods/4f9e9715-f7f1-4407-9a84-f80148af69bc/volumes" Oct 30 20:05:54.158442 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:54.158429 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b0a8a2-ecb8-457f-8c32-5756c3e500d2" path="/var/lib/kubelet/pods/59b0a8a2-ecb8-457f-8c32-5756c3e500d2/volumes" Oct 30 20:05:54.158763 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:54.158752 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffad5dec-cb6f-419e-817d-ab985fb6d823" path="/var/lib/kubelet/pods/ffad5dec-cb6f-419e-817d-ab985fb6d823/volumes" Oct 30 20:05:54.175672 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:54.175644 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:05:59.670264 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:59.670227 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:59.789052 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:05:59.789018 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:00.008337 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:00.008303 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m"] Oct 30 20:06:00.008621 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:00.008590 6514 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" containerID="cri-o://6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942" gracePeriod=30 Oct 30 20:06:00.055809 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:00.055777 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:02.063181 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:02.063144 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:06:04.540887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:04.540008 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndnrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:04.540887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:04.540088 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndnrw" podUID="69def90a-99a6-4c50-a5a0-a9182330ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:04.716037 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:04.715981 6514 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bjr2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:04.716361 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:04.716312 6514 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bjr2f" podUID="688136b4-26ad-4cde-9bc8-c40ef5d7c2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:05.305399 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.305370 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:06:05.383214 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383174 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383395 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383231 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383395 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383264 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383395 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383301 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383395 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383386 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383422 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383488 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:05.383627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383502 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383483 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:05.383627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383548 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.383627 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383585 6514 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x\") pod \"97519e16-b1a9-4c00-aa01-c648ff56de68\" (UID: \"97519e16-b1a9-4c00-aa01-c648ff56de68\") " Oct 30 20:06:05.384066 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383699 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:05.384066 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383709 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:05.384066 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.383942 6514 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-data\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.384066 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.384004 6514 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.384066 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.384025 6514 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-workload-certs\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.384066 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.384041 6514 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-credential-socket\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.384367 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.384112 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:05.386110 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.386085 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:05.386231 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.386101 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:06:05.386305 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.386253 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:05.386305 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.386260 6514 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x" pod "97519e16-b1a9-4c00-aa01-c648ff56de68" (UID: "97519e16-b1a9-4c00-aa01-c648ff56de68"). InnerVolumeSpecName "kube-api-access-44h6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:05.485112 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.485073 6514 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97519e16-b1a9-4c00-aa01-c648ff56de68-istiod-ca-cert\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.485112 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.485112 6514 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44h6x\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-kube-api-access-44h6x\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.485112 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.485123 6514 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-token\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.485335 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.485132 6514 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-podinfo\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.485335 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.485142 6514 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97519e16-b1a9-4c00-aa01-c648ff56de68-istio-envoy\") on node \"ip-10-0-138-233.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:05.552939 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.552845 6514 generic.go:356] "Generic (PLEG): container finished" podID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerID="6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942" exitCode=0 Oct 30 20:06:05.552939 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.552918 6514 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" Oct 30 20:06:05.553424 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.552934 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" event={"ID":"97519e16-b1a9-4c00-aa01-c648ff56de68","Type":"ContainerDied","Data":"6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942"} Oct 30 20:06:05.553424 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.553002 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m" event={"ID":"97519e16-b1a9-4c00-aa01-c648ff56de68","Type":"ContainerDied","Data":"7a28abcfce33d052fe936c7a54aacebc2e606c1177ad5e593447f8c0ad58db85"} Oct 30 20:06:05.553424 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.553038 6514 scope.go:122] "RemoveContainer" containerID="6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942" Oct 30 20:06:05.562371 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.562353 6514 scope.go:122] "RemoveContainer" containerID="2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d" Oct 30 20:06:05.570990 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.570970 6514 scope.go:122] "RemoveContainer" containerID="6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942" Oct 30 20:06:05.571252 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:06:05.571234 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942\": container with ID starting with 6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942 not found: ID does not exist" containerID="6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942" Oct 30 20:06:05.571299 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.571263 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942"} err="failed to get container status \"6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942\": rpc error: code = NotFound desc = could not find container \"6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942\": container with ID starting with 6bc9c459d49f509af1fc6909c9bd1d8c18ed18ff1ad4a55b8bea777143068942 not found: ID does not exist" Oct 30 20:06:05.571299 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.571280 6514 scope.go:122] "RemoveContainer" containerID="2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d" Oct 30 20:06:05.571501 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:06:05.571486 6514 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d\": container with ID starting with 2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d not found: ID does not exist" containerID="2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d" Oct 30 20:06:05.571538 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.571508 6514 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d"} err="failed to get container status \"2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d\": rpc error: code = NotFound desc = could not find container \"2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d\": container with ID starting with 2de38f98ac48b1e7ec0c3b21d72d953bd52a5e54b9ca89deda437ef0f391fe1d not found: ID does not exist" Oct 30 20:06:05.579142 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.579120 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m"] Oct 30 20:06:05.584140 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:05.584114 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-64f2bb-openshift-default-6c68c554d9-dwc8m"] Oct 30 20:06:06.157204 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:06.157168 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" path="/var/lib/kubelet/pods/97519e16-b1a9-4c00-aa01-c648ff56de68/volumes" Oct 30 20:06:07.278258 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:07.278228 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:06:12.568156 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:12.568123 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.673618 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:12.673584 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:13.430348 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:13.430319 6514 scope.go:122] "RemoveContainer" containerID="50f8c56456d73a9c9a0ae71e33c89b8e8ed2bb1fe722c9b2835e6958c47262b2" Oct 30 20:06:13.772273 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:13.772185 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:20.879357 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:06:20.879318 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:06:22.535237 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:22.535204 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:06:49.155722 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:06:49.155685 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2m8b7"] Oct 30 20:06:50.951890 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:06:50.950132 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:07:00.377195 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:00.377086 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d59c5fdcf-dmjs2_d1ff1115-85eb-4812-a46a-5bf2b3582607/manager/0.log" Oct 30 20:07:01.234037 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:01.234005 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bb5597689-4djsx_bc34d546-8598-4841-a3e6-efb6011ac090/manager/0.log" Oct 30 20:07:01.463173 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:01.463143 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x2fwr_350a83c1-ef56-4d86-adc4-e296e3655933/server/0.log" Oct 30 20:07:01.868678 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:01.868648 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-97mp9_9a6f1251-fe5c-4cbf-b306-7db1ef6200b5/proxy/0.log" Oct 30 20:07:02.848435 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:02.848406 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79c7b97c4b-9xh76_d6722016-06cb-4680-a1fa-8737d685e461/maas-api/0.log" Oct 30 20:07:03.211799 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:03.211772 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2aa221a4-c6chx_1ecb9a78-35da-4298-921e-29fc4ce51888/revoke-keys/0.log" Oct 30 20:07:03.394636 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:03.394609 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-q6qps_17ac5339-2d81-4ea2-a8ce-3e61910fa061/revoke-keys/0.log" Oct 30 20:07:03.594922 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:03.594823 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-7jfpm_6809e174-056f-41a8-8c23-4e278fe5bc9b/revoke-keys/0.log" Oct 30 20:07:06.286204 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:06.286177 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pmcfk_8d89664e-71a7-4b99-8761-6b0958fb85d5/kuadrant-console-plugin/0.log" Oct 30 20:07:06.742412 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:06.742380 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2m8b7_90045f0f-21f2-4ce4-b3de-ec9c24c163b9/limitador/0.log" Oct 30 20:07:08.295822 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:08.295797 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-rhxkl_04c85d7a-0eee-4809-831b-654ab8b490f5/kube-auth-proxy/0.log" Oct 30 20:07:08.476176 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:08.476142 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/1.log" Oct 30 20:07:08.658574 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:08.658491 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/1.log" Oct 30 20:07:09.428298 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:09.428255 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/1.log" Oct 30 20:07:09.623616 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:09.623561 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-qbqj6_91f1db0f-3030-450b-8a45-83b100e58823/istio-proxy/0.log" Oct 30 20:07:10.555608 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:10.555577 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v5vhh_04daab4c-1b99-4724-8649-630cdb2b4bc4/payload-pre-processing/0.log" Oct 30 20:07:10.854536 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:10.854450 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 20:07:10.865446 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:10.865423 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p682z_79323c8f-776a-4a3e-942c-a467e8672c53/ovn-acl-logging/0.log" Oct 30 20:07:11.370854 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:11.370822 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/0.log" Oct 30 20:07:11.554928 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:11.554900 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/0.log" Oct 30 20:07:12.297840 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:12.297810 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/0.log" Oct 30 20:07:13.184192 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:13.184160 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz_b4552499-23ca-48dd-9210-1e79e6ad04e6/main/0.log" Oct 30 20:07:13.466485 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:13.466408 6514 scope.go:122] "RemoveContainer" containerID="4a5d56e52c862943c91371dd231abd1b4635ee655d87bce89d097d78c20553b9" Oct 30 20:07:13.541315 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:13.541276 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tlbtd_2c01c154-6ca2-4dcd-b558-2a3790449664/main/0.log" Oct 30 20:07:13.801198 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:13.801110 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:13.904137 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:13.904110 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg_6dda9c25-033d-4e68-bcf8-ba726c31024e/main/0.log" Oct 30 20:07:14.264047 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:14.264017 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7_5d55749f-62cd-4021-b995-5e9245cdbaad/main/0.log" Oct 30 20:07:14.442023 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:14.441993 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb_f37f7bbb-a255-4242-8332-98b97a2a0f68/main/0.log" Oct 30 20:07:20.878827 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:07:20.878787 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:07:24.761673 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:24.761635 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d59c5fdcf-dmjs2_d1ff1115-85eb-4812-a46a-5bf2b3582607/manager/0.log" Oct 30 20:07:25.652175 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:25.652136 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bb5597689-4djsx_bc34d546-8598-4841-a3e6-efb6011ac090/manager/0.log" Oct 30 20:07:25.687431 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:25.687403 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.798878 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:25.798834 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.870320 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:25.870296 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x2fwr_350a83c1-ef56-4d86-adc4-e296e3655933/server/0.log" Oct 30 20:07:26.293299 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:26.293271 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-97mp9_9a6f1251-fe5c-4cbf-b306-7db1ef6200b5/proxy/0.log" Oct 30 20:07:27.104902 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:27.104845 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79c7b97c4b-9xh76_d6722016-06cb-4680-a1fa-8737d685e461/maas-api/0.log" Oct 30 20:07:27.510290 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:27.510251 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2aa221a4-c6chx_1ecb9a78-35da-4298-921e-29fc4ce51888/revoke-keys/0.log" Oct 30 20:07:27.693945 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:27.693908 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-q6qps_17ac5339-2d81-4ea2-a8ce-3e61910fa061/revoke-keys/0.log" Oct 30 20:07:27.869911 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:27.869807 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-7jfpm_6809e174-056f-41a8-8c23-4e278fe5bc9b/revoke-keys/0.log" Oct 30 20:07:30.183943 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:30.183914 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pmcfk_8d89664e-71a7-4b99-8761-6b0958fb85d5/kuadrant-console-plugin/0.log" Oct 30 20:07:30.654582 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:30.654508 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2m8b7_90045f0f-21f2-4ce4-b3de-ec9c24c163b9/limitador/0.log" Oct 30 20:07:31.027007 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:31.026970 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm"] Oct 30 20:07:31.030260 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:31.030234 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7jfpm"] Oct 30 20:07:32.075704 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:32.075677 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-rhxkl_04c85d7a-0eee-4809-831b-654ab8b490f5/kube-auth-proxy/0.log" Oct 30 20:07:32.165547 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:32.165513 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6809e174-056f-41a8-8c23-4e278fe5bc9b" path="/var/lib/kubelet/pods/6809e174-056f-41a8-8c23-4e278fe5bc9b/volumes" Oct 30 20:07:32.270253 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:32.270222 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/1.log" Oct 30 20:07:32.457496 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:32.457473 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/1.log" Oct 30 20:07:33.227984 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:33.227954 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/1.log" Oct 30 20:07:33.426153 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:33.426121 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-qbqj6_91f1db0f-3030-450b-8a45-83b100e58823/istio-proxy/0.log" Oct 30 20:07:34.470249 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:34.470221 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v5vhh_04daab4c-1b99-4724-8649-630cdb2b4bc4/payload-pre-processing/0.log" Oct 30 20:07:35.402384 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:35.402356 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-cwnpz_b4552499-23ca-48dd-9210-1e79e6ad04e6/main/0.log" Oct 30 20:07:35.811459 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:35.811429 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tlbtd_2c01c154-6ca2-4dcd-b558-2a3790449664/main/0.log" Oct 30 20:07:36.184080 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:36.184055 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2tdg_6dda9c25-033d-4e68-bcf8-ba726c31024e/main/0.log" Oct 30 20:07:36.556403 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:36.556328 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pf8q7_5d55749f-62cd-4021-b995-5e9245cdbaad/main/0.log" Oct 30 20:07:36.737927 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:36.737895 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2xgsb_f37f7bbb-a255-4242-8332-98b97a2a0f68/main/0.log" Oct 30 20:07:48.024619 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:48.024577 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps"] Oct 30 20:07:48.027566 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:48.027528 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-q6qps"] Oct 30 20:07:48.156681 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:07:48.156655 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ac5339-2d81-4ea2-a8ce-3e61910fa061" path="/var/lib/kubelet/pods/17ac5339-2d81-4ea2-a8ce-3e61910fa061/volumes" Oct 30 20:07:50.881890 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:07:50.881826 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:08:06.025325 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:06.025287 6514 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx"] Oct 30 20:08:06.028725 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:06.028698 6514 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa221a4-c6chx"] Oct 30 20:08:06.158143 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:06.158112 6514 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecb9a78-35da-4298-921e-29fc4ce51888" path="/var/lib/kubelet/pods/1ecb9a78-35da-4298-921e-29fc4ce51888/volumes" Oct 30 20:08:08.044429 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:08.044384 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:13.862978 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:13.862943 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:15.611371 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:15.611339 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bb5597689-4djsx_bc34d546-8598-4841-a3e6-efb6011ac090/manager/0.log" Oct 30 20:08:15.879482 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:15.879406 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bb5597689-4djsx_bc34d546-8598-4841-a3e6-efb6011ac090/manager/0.log" Oct 30 20:08:20.879907 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:08:20.879841 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:08:47.907731 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:47.907686 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:50.880455 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:08:50.880425 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:08:59.186956 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:08:59.186921 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:02.963371 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:02.963340 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bb5597689-4djsx_bc34d546-8598-4841-a3e6-efb6011ac090/manager/0.log" Oct 30 20:09:03.216040 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:03.215960 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bb5597689-4djsx_bc34d546-8598-4841-a3e6-efb6011ac090/manager/0.log" Oct 30 20:09:04.426137 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426100 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9glm/must-gather-fbjq7"] Oct 30 20:09:04.426530 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426510 6514 cpu_manager.go:414] "containerMap: removing container" podUID="ffad5dec-cb6f-419e-817d-ab985fb6d823" containerName="payload-pre-processing" Oct 30 20:09:04.426530 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426521 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffad5dec-cb6f-419e-817d-ab985fb6d823" containerName="payload-pre-processing" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426537 6514 cpu_manager.go:414] "containerMap: removing container" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426543 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426551 6514 cpu_manager.go:414] "containerMap: removing container" podUID="59b0a8a2-ecb8-457f-8c32-5756c3e500d2" containerName="payload-processing" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426557 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b0a8a2-ecb8-457f-8c32-5756c3e500d2" containerName="payload-processing" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426566 6514 cpu_manager.go:414] "containerMap: removing container" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426570 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426578 6514 cpu_manager.go:414] "containerMap: removing container" podUID="4f9e9715-f7f1-4407-9a84-f80148af69bc" containerName="maas-api" Oct 30 20:09:04.426612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426583 6514 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9e9715-f7f1-4407-9a84-f80148af69bc" containerName="maas-api" Oct 30 20:09:04.426891 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426643 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffad5dec-cb6f-419e-817d-ab985fb6d823" containerName="payload-pre-processing" Oct 30 20:09:04.426891 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426654 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" Oct 30 20:09:04.426891 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426663 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b0a8a2-ecb8-457f-8c32-5756c3e500d2" containerName="payload-processing" Oct 30 20:09:04.426891 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426670 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="97519e16-b1a9-4c00-aa01-c648ff56de68" containerName="istio-proxy" Oct 30 20:09:04.426891 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.426676 6514 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9e9715-f7f1-4407-9a84-f80148af69bc" containerName="maas-api" Oct 30 20:09:04.430147 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.430127 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.434240 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.434220 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9glm\"/\"kube-root-ca.crt\"" Oct 30 20:09:04.435924 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.435903 6514 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9glm\"/\"openshift-service-ca.crt\"" Oct 30 20:09:04.435924 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.435913 6514 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x9glm\"/\"default-dockercfg-fzhm4\"" Oct 30 20:09:04.444007 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.443985 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/must-gather-fbjq7"] Oct 30 20:09:04.451178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.451154 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:04.521146 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.521117 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c74a78d-8a27-4a47-8d03-63a781cd9c44-must-gather-output\") pod \"must-gather-fbjq7\" (UID: \"4c74a78d-8a27-4a47-8d03-63a781cd9c44\") " pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.521302 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.521261 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9cc\" (UniqueName: \"kubernetes.io/projected/4c74a78d-8a27-4a47-8d03-63a781cd9c44-kube-api-access-2k9cc\") pod \"must-gather-fbjq7\" (UID: \"4c74a78d-8a27-4a47-8d03-63a781cd9c44\") " pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.598396 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.598363 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:04.622761 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.622730 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9cc\" (UniqueName: \"kubernetes.io/projected/4c74a78d-8a27-4a47-8d03-63a781cd9c44-kube-api-access-2k9cc\") pod \"must-gather-fbjq7\" (UID: \"4c74a78d-8a27-4a47-8d03-63a781cd9c44\") " pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.623042 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.622801 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c74a78d-8a27-4a47-8d03-63a781cd9c44-must-gather-output\") pod \"must-gather-fbjq7\" (UID: \"4c74a78d-8a27-4a47-8d03-63a781cd9c44\") " pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.623209 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.623190 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c74a78d-8a27-4a47-8d03-63a781cd9c44-must-gather-output\") pod \"must-gather-fbjq7\" (UID: \"4c74a78d-8a27-4a47-8d03-63a781cd9c44\") " pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.632043 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.632017 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9cc\" (UniqueName: \"kubernetes.io/projected/4c74a78d-8a27-4a47-8d03-63a781cd9c44-kube-api-access-2k9cc\") pod \"must-gather-fbjq7\" (UID: \"4c74a78d-8a27-4a47-8d03-63a781cd9c44\") " pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:04.745330 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:04.745307 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/must-gather-fbjq7" Oct 30 20:09:05.016842 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:05.016818 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/must-gather-fbjq7"] Oct 30 20:09:05.018948 ip-10-0-138-233 kubenswrapper[6514]: W1030 20:09:05.018917 6514 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c74a78d_8a27_4a47_8d03_63a781cd9c44.slice/crio-c8fd7b21f266f2c6cef569499eedc19d1c47d8c9fc3f5b64c4ab062780a00326 WatchSource:0}: Error finding container c8fd7b21f266f2c6cef569499eedc19d1c47d8c9fc3f5b64c4ab062780a00326: Status 404 returned error can't find the container with id c8fd7b21f266f2c6cef569499eedc19d1c47d8c9fc3f5b64c4ab062780a00326 Oct 30 20:09:05.020851 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:05.020835 6514 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:09:05.537063 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:05.537019 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/must-gather-fbjq7" event={"ID":"4c74a78d-8a27-4a47-8d03-63a781cd9c44","Type":"ContainerStarted","Data":"c8fd7b21f266f2c6cef569499eedc19d1c47d8c9fc3f5b64c4ab062780a00326"} Oct 30 20:09:06.546691 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:06.546650 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/must-gather-fbjq7" event={"ID":"4c74a78d-8a27-4a47-8d03-63a781cd9c44","Type":"ContainerStarted","Data":"54c464027cd7ef91ce485527e46c77562d6436068e01799724655b9f232960fd"} Oct 30 20:09:06.547122 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:06.546703 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/must-gather-fbjq7" event={"ID":"4c74a78d-8a27-4a47-8d03-63a781cd9c44","Type":"ContainerStarted","Data":"600675fe2d0d50f0b221347fedecf44977842ae2d0108f7215836771cbcdc5a4"} Oct 30 20:09:06.568027 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:06.567977 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9glm/must-gather-fbjq7" podStartSLOduration=1.7970557280000001 podStartE2EDuration="2.567960377s" totalImagesPullingTime="770.904649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:06.566088964 +0000 UTC m=+3116.523908147" watchObservedRunningTime="2026-10-30 20:09:06.567960377 +0000 UTC m=+3116.525779568" Oct 30 20:09:07.790883 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:07.790829 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5sqtd_6b654842-b7a7-4e8d-8c92-8729111d0373/global-pull-secret-syncer/0.log" Oct 30 20:09:08.044582 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:08.044358 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" Oct 30 20:09:08.207513 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:08.207486 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-233.us-east-2.compute.internal_2cb181debb656c62bb70f1078166094d/haproxy/0.log" Oct 30 20:09:13.501887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:13.498591 6514 scope.go:122] "RemoveContainer" containerID="448d1bb0fe6fb9ebf760feb8b1934dad9de35d9a5377fc5296203fccae5b038a" Oct 30 20:09:13.522134 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:13.519806 6514 scope.go:122] "RemoveContainer" containerID="b431fb8bc7dc32b780f7672732d1832ae7f1debedc1859ca75280c94a616bee2" Oct 30 20:09:13.562065 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:13.561778 6514 scope.go:122] "RemoveContainer" containerID="5b7d50196b9906210e470bf1b1d883f99872cdae848b9088477bb25fe7c638fd" Oct 30 20:09:13.947801 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:13.947728 6514 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ssdd_26bc4ca2-8079-4560-8891-dc52f796c72f/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:15.064724 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:15.064691 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pmcfk_8d89664e-71a7-4b99-8761-6b0958fb85d5/kuadrant-console-plugin/0.log" Oct 30 20:09:15.215612 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:15.215579 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2m8b7_90045f0f-21f2-4ce4-b3de-ec9c24c163b9/limitador/0.log" Oct 30 20:09:16.623724 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.623696 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/alertmanager/0.log" Oct 30 20:09:16.657695 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.657667 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/config-reloader/0.log" Oct 30 20:09:16.694518 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.694482 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/kube-rbac-proxy-web/0.log" Oct 30 20:09:16.721620 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.721577 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/kube-rbac-proxy/0.log" Oct 30 20:09:16.757717 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.757680 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/kube-rbac-proxy-metric/0.log" Oct 30 20:09:16.785899 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.785853 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/prom-label-proxy/0.log" Oct 30 20:09:16.827638 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.827558 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b621bd4-d341-4a35-a0f5-73dcb25faae3/init-config-reloader/0.log" Oct 30 20:09:16.928084 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.928017 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-277gf_6fb8c089-70f4-418d-8cd4-fd2393997ccf/kube-state-metrics/0.log" Oct 30 20:09:16.952581 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.952547 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-277gf_6fb8c089-70f4-418d-8cd4-fd2393997ccf/kube-rbac-proxy-main/0.log" Oct 30 20:09:16.990100 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:16.990069 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-277gf_6fb8c089-70f4-418d-8cd4-fd2393997ccf/kube-rbac-proxy-self/0.log" Oct 30 20:09:17.034855 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.034819 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8988ff5d4-t8x89_937e5d34-bca8-4f99-bb4a-22a62c63fe73/metrics-server/0.log" Oct 30 20:09:17.075165 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.075128 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-slbxc_bfa4a876-df8d-45b5-aa59-37bf104db694/monitoring-plugin/0.log" Oct 30 20:09:17.130623 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.130589 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftfsr_faee8bf1-c317-4506-a5a7-e7317ab3909d/node-exporter/0.log" Oct 30 20:09:17.175896 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.175846 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftfsr_faee8bf1-c317-4506-a5a7-e7317ab3909d/kube-rbac-proxy/0.log" Oct 30 20:09:17.225252 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.225222 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftfsr_faee8bf1-c317-4506-a5a7-e7317ab3909d/init-textfile/0.log" Oct 30 20:09:17.552622 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.552545 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xfsnz_da476d5e-0b9c-428d-a53a-98f99d2b15c5/kube-rbac-proxy-main/0.log" Oct 30 20:09:17.583413 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.583377 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xfsnz_da476d5e-0b9c-428d-a53a-98f99d2b15c5/kube-rbac-proxy-self/0.log" Oct 30 20:09:17.615512 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.615481 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xfsnz_da476d5e-0b9c-428d-a53a-98f99d2b15c5/openshift-state-metrics/0.log" Oct 30 20:09:17.641665 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.641622 6514 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs"] Oct 30 20:09:17.649766 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.649736 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.662996 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.662194 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs"] Oct 30 20:09:17.698317 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.698273 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-podres\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.698519 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.698339 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkcq\" (UniqueName: \"kubernetes.io/projected/f809e06a-34f9-4007-be26-64c5dfa27dcf-kube-api-access-gwkcq\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.698519 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.698371 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-lib-modules\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.698519 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.698409 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-proc\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.698519 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.698461 6514 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-sys\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800140 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800103 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-proc\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800339 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800217 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-sys\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800339 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800256 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-sys\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800339 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800223 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-proc\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800537 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800276 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-podres\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800537 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800368 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-podres\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800537 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800418 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkcq\" (UniqueName: \"kubernetes.io/projected/f809e06a-34f9-4007-be26-64c5dfa27dcf-kube-api-access-gwkcq\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800537 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800465 6514 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-lib-modules\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.800727 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.800588 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f809e06a-34f9-4007-be26-64c5dfa27dcf-lib-modules\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.809851 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.809791 6514 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkcq\" (UniqueName: \"kubernetes.io/projected/f809e06a-34f9-4007-be26-64c5dfa27dcf-kube-api-access-gwkcq\") pod \"perf-node-gather-daemonset-2mscs\" (UID: \"f809e06a-34f9-4007-be26-64c5dfa27dcf\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:17.974325 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:17.974080 6514 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:18.086982 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.086904 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78bcbbb9fb-frn5x_f438e609-1285-4d67-8622-7b5022bdee31/telemeter-client/0.log" Oct 30 20:09:18.124040 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.123996 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78bcbbb9fb-frn5x_f438e609-1285-4d67-8622-7b5022bdee31/reload/0.log" Oct 30 20:09:18.159953 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.159926 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78bcbbb9fb-frn5x_f438e609-1285-4d67-8622-7b5022bdee31/kube-rbac-proxy/0.log" Oct 30 20:09:18.277386 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.277350 6514 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs"] Oct 30 20:09:18.717203 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.717162 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" event={"ID":"f809e06a-34f9-4007-be26-64c5dfa27dcf","Type":"ContainerStarted","Data":"5ef55dd638f4289548cf5606757711fd3628e6d90f95c570356699e000ce0168"} Oct 30 20:09:18.717628 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.717221 6514 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" event={"ID":"f809e06a-34f9-4007-be26-64c5dfa27dcf","Type":"ContainerStarted","Data":"8164c2b39d70d584da8d6345c62cf25cde089b8015496fa2ab0979f3ff50540e"} Oct 30 20:09:18.717628 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.717257 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:18.737926 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:18.737840 6514 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" podStartSLOduration=1.737826278 podStartE2EDuration="1.737826278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:18.736270318 +0000 UTC m=+3128.694089510" watchObservedRunningTime="2026-10-30 20:09:18.737826278 +0000 UTC m=+3128.695645467" Oct 30 20:09:20.810596 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:20.810560 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-588db67665-hvtxk_9444c2ce-87b9-4697-935c-9e783689b2a4/console/0.log" Oct 30 20:09:20.886061 ip-10-0-138-233 kubenswrapper[6514]: E1030 20:09:20.886014 6514 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54p6s" containerName="extractor" Oct 30 20:09:22.791893 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:22.791850 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fjmnw_3c9c1b81-6bc0-406b-a0c1-a370afa909b0/dns/0.log" Oct 30 20:09:22.815996 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:22.815970 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fjmnw_3c9c1b81-6bc0-406b-a0c1-a370afa909b0/kube-rbac-proxy/0.log" Oct 30 20:09:22.929447 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:22.929414 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p74tq_1b6cc8cd-4e1e-472a-97a1-75b4bf3f01ab/dns-node-resolver/0.log" Oct 30 20:09:23.684887 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:23.684806 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkrdd_1df29a6a-001e-463a-9a07-0cfe1c0fe6a0/node-ca/0.log" Oct 30 20:09:24.738229 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:24.738201 6514 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-2mscs" Oct 30 20:09:27.166127 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.166092 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-rhxkl_04c85d7a-0eee-4809-831b-654ab8b490f5/kube-auth-proxy/0.log" Oct 30 20:09:27.200350 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.200319 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/0.log" Oct 30 20:09:27.200565 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.200527 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/1.log" Oct 30 20:09:27.235390 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.235352 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/1.log" Oct 30 20:09:27.236024 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.236000 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/0.log" Oct 30 20:09:27.373597 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.373559 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/0.log" Oct 30 20:09:27.374447 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.374420 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/1.log" Oct 30 20:09:27.402735 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.402685 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-qbqj6_91f1db0f-3030-450b-8a45-83b100e58823/istio-proxy/0.log" Oct 30 20:09:27.562660 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:27.562620 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v5vhh_04daab4c-1b99-4724-8649-630cdb2b4bc4/payload-pre-processing/0.log" Oct 30 20:09:29.626901 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:29.626856 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bpk2j_471d5763-db1f-46f7-99fe-d55f45fa9a76/serve-healthcheck-canary/0.log" Oct 30 20:09:29.934648 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:29.934618 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-rhxkl_04c85d7a-0eee-4809-831b-654ab8b490f5/kube-auth-proxy/0.log" Oct 30 20:09:29.957178 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:29.957149 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/0.log" Oct 30 20:09:29.959026 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:29.959000 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-82vrf_88856f6f-b914-4487-9da6-5107913dc38e/istio-proxy/1.log" Oct 30 20:09:29.976562 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:29.976541 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/1.log" Oct 30 20:09:29.976714 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:29.976674 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bjr2f_688136b4-26ad-4cde-9bc8-c40ef5d7c2fa/istio-proxy/0.log" Oct 30 20:09:30.040520 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.040490 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/0.log" Oct 30 20:09:30.041886 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.041847 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ndnrw_69def90a-99a6-4c50-a5a0-a9182330ff80/istio-proxy/1.log" Oct 30 20:09:30.055732 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.055705 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-qbqj6_91f1db0f-3030-450b-8a45-83b100e58823/istio-proxy/0.log" Oct 30 20:09:30.143086 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.143049 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v5vhh_04daab4c-1b99-4724-8649-630cdb2b4bc4/payload-pre-processing/0.log" Oct 30 20:09:30.143086 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.143079 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54p6s_c346bfd4-9367-4a93-a44e-022b444c9f8c/kube-rbac-proxy/0.log" Oct 30 20:09:30.168809 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.168780 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54p6s_c346bfd4-9367-4a93-a44e-022b444c9f8c/exporter/0.log" Oct 30 20:09:30.199392 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:30.199306 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54p6s_c346bfd4-9367-4a93-a44e-022b444c9f8c/extractor/0.log" Oct 30 20:09:33.012901 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:33.012832 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-68fb5c5984-7qpdv_c9ee772c-a7b0-469f-bc85-bcc6c5d74a91/manager/0.log" Oct 30 20:09:33.033693 ip-10-0-138-233 kubenswrapper[6514]: I1030 20:09:33.033669 6514 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-68fb5c5984-7qpdv_c9ee772c-a7b0-469f-bc85-bcc6c5d74a91/copy-manifests/0.log"